From 44db45ac18da3d7516553954d52d3705d3889575 Mon Sep 17 00:00:00 2001 From: schickling-assistant <261620128+schickling-assistant@users.noreply.github.com> Date: Wed, 26 Aug 2026 17:34:29 +0200 Subject: [PATCH 1/2] feat(observation): compose driver and PTY session fidelity A deployed 627-seat consumer exposed the envelope's coverage hole: readers were fleet-wide while producers existed only when an st2 driver owned the session. Compose a launcher-agnostic session projection at read time from PTY lastOutputAtMs, with explicit fidelity=driver|session. Definite fresh driver state wins. Missing/unknown driver state falls back to alive PTY output activity; session fidelity proves only state+since. No coarse record writer, fencing, heartbeat, or launcher awareness is introduced. Includes the user-confirmed constitutional VRS amendment: requirements, spec, ontology, decision 0006 Amendment 1, and measured experiment. agent-identity: unknown agent-persona: generalist agent-supervisor: unavailable agent-tool: OMP agent-tool-version: 18.0.3 agent-runtime: OMP 18.0.3 tooling-profile: dotfiles@e4789b0 --- ...tate-is-a-driver-written-catalog-record.md | 30 +++ ...6-launcher-independent-session-activity.md | 80 ++++++ docs/vrs/05-harness-state/requirements.md | 57 ++++- docs/vrs/05-harness-state/spec.md | 126 +++++++--- docs/vrs/ontology.md | 18 +- src/agents.rs | 236 ++++++++++++++++-- src/harness_state.rs | 23 ++ src/main.rs | 24 +- tests/doctor.rs | 8 +- 9 files changed, 519 insertions(+), 83 deletions(-) create mode 100644 docs/vrs/05-harness-state/.experiments/2026-08-26-launcher-independent-session-activity.md diff --git a/docs/vrs/.decisions/0006-observed-harness-state-is-a-driver-written-catalog-record.md b/docs/vrs/.decisions/0006-observed-harness-state-is-a-driver-written-catalog-record.md index bd45fff4..b0a9831e 100644 --- a/docs/vrs/.decisions/0006-observed-harness-state-is-a-driver-written-catalog-record.md +++ b/docs/vrs/.decisions/0006-observed-harness-state-is-a-driver-written-catalog-record.md @@ -108,3 +108,33 @@ scoping delivery-input watching is a prerequisite, not a precaution. - Root DQ2 and DQ3 are updated: pi gains an evented signal, the observed half of DQ3 is specified here, and the declared half (activity status, plan, plan step) plus supervisor-following behavior remain open. + +## Amendment 1 — PTY session fidelity composes at read time + +Accepted by Johannes on 2026-08-26 after deploying this decision's original +implementation to a 627-seat downstream catalog. The envelope and readers were +live while zero seats produced a record: the deployment launched harnesses +outside st2's native drivers, and nothing enforced or measured the assumption +that a driver owned every session. The original decision remains correct for +the **fine driver record** and wrong as the only coverage mechanism. + +Observed harness state therefore gains an additive +`fidelity ∈ driver | session` discriminator and a launcher-agnostic session +projection: + +- `driver` retains this decision's full record, fencing, freshness, and + complete tuple semantics; +- `session` is a read-time projection over the canonical PTY session's + persisted `lastOutputAtMs`, covering only `state` and `since`; its + blocked/input/ask axes remain `unknown`; +- a fresh definite driver observation wins; session fidelity replaces a + missing or derived-unknown driver observation; +- the session projection writes no `harness-state` record and creates no + second writer class. + +The PTY daemon is the observer because it already processes every output byte. +st2 does not know or branch on the launcher — the canonical agent task's PTY id +is the bus id regardless of whether the child is axe, a native st2 driver, or a +future launcher. The decision rejects `pty stats` polling, scrollback deltas, +and event-stream following based on the fleet measurements in +[`05-harness-state/.experiments/2026-08-26-launcher-independent-session-activity.md`](../05-harness-state/.experiments/2026-08-26-launcher-independent-session-activity.md). diff --git a/docs/vrs/05-harness-state/.experiments/2026-08-26-launcher-independent-session-activity.md b/docs/vrs/05-harness-state/.experiments/2026-08-26-launcher-independent-session-activity.md new file mode 100644 index 00000000..9d9d4cbc --- /dev/null +++ b/docs/vrs/05-harness-state/.experiments/2026-08-26-launcher-independent-session-activity.md @@ -0,0 +1,80 @@ +# Launcher-independent PTY session activity + +2026-08-26, Linux, a downstream catalog with 627 declared seats and 60 live PTY sessions during the measurement. The investigation followed a deployed observed-harness-state envelope whose reader was live in st2 and fractal while every seat still returned `observedState: null`. + +## Question + +What is the lowest-global-complexity, launcher-agnostic source of coarse harness activity for every managed session, and can it remain efficient in a busy 627-seat catalog? + +## Method + +The investigation traced the deployed producer/read paths in st2, identified the process refreshing presence, inspected the PTY daemon's output and persistence paths, measured the existing candidate surfaces, and compared their asymptotic and measured fleet costs. + +Reproduction commands: + +```sh +# Current-scale cost and live-session count +time pty stats --json > stats.json +jq 'length' stats.json + +# Persisted registry shape and event distribution +jq 'keys' /.json +jq -r '.type' /.events.jsonl | sort | uniq -c | sort -rn + +# Direct-read baseline (single process) +time jq -s 'length' /*.json >/dev/null + +# Source contracts +rg -n 'scrollbackUsed|scrollbackCapacity|ptyProcess.onData' src/server.ts +``` + +## Result + +**Coverage was conditional on the launch path, not the harness inventory.** The deployed catalog launched harnesses through an external wrapper. st2's rich producers run only inside st2's native session drivers (or their hook/channel siblings), while the harness-blind ding sidecar refreshed presence for every live seat. `st2 hooks verify`, presence, and catalog checks were green with zero `harness-state` records. A launcher may therefore adopt the roster reader without any producer; nothing measured that gap. + +**The PTY daemon is the universal observer.** Its `onData` handler already receives every PTY output chunk before feeding xterm-headless and clients. Stamping a timestamp there is O(1) and adds no observer, stream, process, or harness/launcher coupling. + +**Terminal-buffer deltas are not an activity clock.** `pty stats` reports `scrollbackUsed = buf.length` and capacity `rows + scrollback` (`src/server.ts`). The buffer is bounded; once full, its length stops advancing while output continues. Deriving activity from length deltas therefore fails systematically on the longest-running sessions. + +**The existing event stream is sparse, not an output stream.** The three largest sampled event logs carried 840–961 records and were 99% `title_change`; only a few `user.agent.status`, lifecycle, bell, or cursor events appeared. Harnesses and launchers may emit useful semantic edges, but absence of an event is not evidence of idle output. + +**Shelling to `pty stats` is too expensive at fleet scale.** One bulk `pty stats --json` snapshot took 520 ms for 60 sessions (~8.7 ms/session), projecting to ~5.5 s for 627 sessions. The cost includes process/resource probes that observed-state composition does not need. + +**Direct metadata joins are cheap.** Reading 300 persisted session JSON files took 19 ms in one process (and 379 ms in the deliberately worst process-per-file form). A native Rust reader over the small files is well below the interactive roster budget. No subprocess is required. + +**The implemented composed roster remains sub-second at full declared-fleet +scale.** A synthetic catalog with 627 local, live sessions (627 pid probes and +627 metadata reads; half stamped 500 ms ago, half 120 s ago) ran +`st2 agents --json` ten times after one warm-up: 361.89 ms minimum, 394.39 ms +median, 551.58 ms maximum, 412.64 ms mean. The result contained exactly +314 `active` and 313 `idle` session-fidelity observations. The benchmark used +the debug binary, so it is a conservative bound rather than a release-build +claim. + +## Conclusion + +The global minimum-complexity shape is: + +```text +PTY output -> daemon lastOutputAtMs stamp (O(1)/chunk) + -> locked session metadata persist (trailing debounce <= 1/s) + -> st2 read-time join (alive + recent output => active; alive + older => idle) + -> fresh definite driver observation takes precedence +``` + +The coarse session projection does not write `harness-state`, so it introduces no writer identity, fencing, heartbeat, history, or retention contract. It is launcher- and harness-agnostic. `fidelity = session | driver` tells consumers which axes are proved; session fidelity covers only `state` and `since`. + +The implemented benchmark covered all 627 sessions simultaneously live, with +the liveness and metadata join active for every row. The one-second metadata +debounce is a write-amplification bound, not an activity threshold; st2 owns +the 60-second activity window and 30-second future-skew guard. These constants +require tuning only if captured turn streams show maintained harnesses going +silent for longer than the window while still actively producing a turn. + +## VRS Impact + +- Amend decision 0006: the driver-written record remains the fine layer, not the only coverage mechanism. +- Add OHS-A04 and OHS-R11–R13: PTY evidence, read-time projection, precedence, and fleet cost. +- Extend OHS-R09 with `observedState.fidelity = driver | session`. +- Update the spec's overview, exposure wire, verification plan, and ontology. +- Leave DQ-H5 (remote supervisor semantics) open; session fidelity is deliberately same-host because PTY metadata is host-local. diff --git a/docs/vrs/05-harness-state/requirements.md b/docs/vrs/05-harness-state/requirements.md index ee0834ef..0796bfd2 100644 --- a/docs/vrs/05-harness-state/requirements.md +++ b/docs/vrs/05-harness-state/requirements.md @@ -41,6 +41,12 @@ path reads this record. session's driver processes — the wrapper that owns the presence lease, and the channel or hook subprocesses it shares its incarnation token with; nothing verifies that claim. +- **OHS-A04 PTY output is universal session evidence:** Every maintained + launcher runs the agent task in the declared PTY session. The PTY daemon + necessarily observes every output byte to maintain terminal state, regardless + of which launcher or harness produced it. This is sufficient evidence for a + coarse `active | idle` classification, but proves nothing about `blockedOn`, + `ask`, or `inputBuffer`. ## Acceptable Tradeoffs @@ -106,7 +112,7 @@ path reads this record. not the agent directory wholesale. This is a prerequisite: the record sits in a tree the Codex pump watches unfiltered today. -### Must be produced by drivers under the evidence rule +### Must derive from positive evidence - **OHS-R05 Driver-owned projection:** Classification is driver work. The Codex producer projects the existing control state with the corrected rows: @@ -133,11 +139,11 @@ path reads this record. narrowing, not a closure: what it cannot prove, the relaunch-time written claim supersedes and the staleness horizon bounds. A fresh `ended` survives the check: a terminal record is supposed to outlive its writer. -- **OHS-R08 All-harness coverage:** Codex, Claude, pi, and OpenCode each ship - a producer. pi's is evented through the injected extension (the positive - idle signal root `DQ2` asks for). OpenCode reaches driver parity first — - typed driver, session wrapper owning the presence lease, then its producer - and native delivery transport. +- **OHS-R08 All-harness coverage:** Every local running agent has session + fidelity independent of its launcher or harness. Codex, Claude, pi, + OpenCode, and OMP additionally ship fine driver producers for sessions the + corresponding st2 driver owns. Fine coverage may vary by launch path; + session coverage may not. ### Must be readable beside declared presence @@ -145,11 +151,32 @@ path reads this record. declared `status` in one payload — the wedged-agent comparison (declared `busy`, observed `idle`) must not require joining two commands. Observed state is a third independent axis: it never rewrites presence, desired - lifecycle, or `lastActivity`, and the pinned roster wire assertions change + lifecycle, or `lastActivity`. `observedState.fidelity ∈ driver | session` + tells consumers which axes are proved: driver fidelity covers the full + tuple; session fidelity covers `state` only and leaves `blockedOn`, `ask`, + and `inputBuffer` `unknown`. The pinned roster wire assertions change deliberately, in the same change, with the new proof named. -- **OHS-R10 Doctor exposure:** Doctor surfaces observed state for agents it - owns as advisory output — a stale or session-dead record beside a `running` - desired state is worth a warning, never an exit-code failure in v1. +- **OHS-R10 Doctor exposure:** Doctor surfaces composed observed state for + agents it owns as advisory output — fidelity and an indeterminate reason are + explicit; absence names a missing driver record *and* missing PTY activity + stamp. None is an exit-code failure in v1. +- **OHS-R11 Launcher-agnostic session projection:** The PTY daemon stamps + `lastOutputAtMs` while processing output and persists it to the canonical + session metadata, debounced to at most one metadata write per second per busy + session. st2 joins that metadata at read time using the canonical agent task + mapping `pty_id = bus_id`: alive plus output inside the activity window + derives session-fidelity `active`; alive plus older output derives + session-fidelity `idle`; missing liveness or output evidence derives nothing. + st2 never branches on, imports, or names the launcher. +- **OHS-R12 Fine-over-session precedence:** A definite fresh driver record + wins over session activity. A missing or derived-`unknown` driver record + falls back to session activity. The session projection never becomes a + `harness-state` writer and therefore introduces no writer identity, fencing, + heartbeat, or record-retention contract. +- **OHS-R13 Bounded fleet cost:** Output stamping is O(1) in the PTY daemon's + existing parse path. The persist debounce bounds write amplification. + Roster reads consume the small per-session metadata directly; they do not + shell out to `pty stats`, follow event streams, or scan terminal buffers. ## Evidence @@ -162,3 +189,13 @@ the Codex `activeFlags` schema present on all supported codex-cli versions machine and its hold reasons, the unfiltered agent-dir watch beside the presence refresh that writes into it, and `src/harness_state.rs`, which implements the envelope this file ratifies. + +The session-fidelity measurements were taken 2026-08-26 on a 627-seat +downstream catalog: the shipped envelope had zero producer records because the +launcher bypassed st2 drivers; `pty stats --json` cost 520 ms for 60 sessions +(~5.5 s projected to 627); `scrollbackUsed` is the bounded terminal-buffer +length and saturates; the PTY event log is sparse and title-change dominated; +reading 300 persisted session metadata files cost 19 ms in one process. These +rule out stats polling, scrollback deltas, and event-following in favor of one +daemon stamp plus a direct read-time join. See +[`05-harness-state/.experiments/2026-08-26-launcher-independent-session-activity.md`](./.experiments/2026-08-26-launcher-independent-session-activity.md). diff --git a/docs/vrs/05-harness-state/spec.md b/docs/vrs/05-harness-state/spec.md index 8db236eb..5d828d72 100644 --- a/docs/vrs/05-harness-state/spec.md +++ b/docs/vrs/05-harness-state/spec.md @@ -17,32 +17,35 @@ are tracked in [open-questions.md](./open-questions.md). ## Scope -This specification defines the record, its freshness and derivation rules, the -per-harness producers, and the roster/Doctor exposure. It does not define: -transition history or a `--watch` surface (deferred, OHS-T03); idle thresholds, -escalation, or notification policy (#173's, per root `R20`); the withdrawn -host-local hot tier; the cut PTY screen observer; or any `AGENT-SPEC.md` -change (separate authority, root `DQ3`). +This specification defines the fine driver record, its freshness and +derivation rules, the per-harness producers, the launcher-agnostic PTY session +projection, their precedence, and the roster/Doctor exposure. It does not +define: transition history or a `--watch` surface (deferred, OHS-T03); +escalation or notification policy (#173's, per root `R20`); the withdrawn +host-local screen-classification observer; or any `AGENT-SPEC.md` change +(separate authority, root `DQ3`). ## Overview ```text - codex app-server claude hooks + pi extension opencode - control stream wrapper child poll (evented) server surface - | | | | - v v v v - [driver-owned projection: idle/active/child/ended × blockedOn (+ask) × inputBuffer] - | - v declared axis (unchanged) - /harness-state /status - st2.harness-state.v1 presence, agent-authored - transition writes + 5-min heartbeat | - | | - +--------------------+-----------------------------+ - v - st2 agents --json (status ∥ observedState) - st2 doctor (advisory) - downstream TUI (spinner, blocked-on-you) + fine fidelity (driver-owned) session fidelity (launcher-agnostic) + + codex/claude/pi/opencode/OMP drivers PTY daemon already parses output bytes + | | + v v + /harness-state /.json + st2.harness-state.v1 lastOutputAtMs (persist ≤1/s) + full tuple + fencing + heartbeat | + | | + +--------------+-----------------------+ + v + roster read-time fold + fresh definite driver > PTY session activity + | + v + st2 agents --json (status ∥ observedState.fidelity) + st2 doctor (advisory) + downstream TUI (fine semantics or coarse active/idle) ``` ## Record (OHS-R01, OHS-R03) @@ -364,12 +367,47 @@ pending. The status seed trusts exactly the pinned words (`busy`, `retry`, pre-signal escalation cover with the exit the grace-window reap actually observed. +## PTY session projection (OHS-R08, OHS-R11–R13) + +The canonical agent task's PTY id is its host-qualified bus id. This is the +runner's authored mapping, not a convention recovered from a launcher. The PTY +daemon stamps the unix-millisecond time of each output chunk in memory while +feeding the same chunk to the terminal emulator. A trailing-edge one-second +debounce persists the newest value as `lastOutputAtMs` through PTY's locked +metadata mutation. Exit metadata carries the final in-memory stamp. + +Roster reads on the local host: + +1. read and derive the driver record with its existing liveness probe; +2. return a definite driver observation unchanged; +3. otherwise prove the canonical PTY session alive from its pidfile and read + `/.json`; +4. derive session-fidelity `active` when `lastOutputAtMs` is no more than 60 s + old, `idle` when older, `unknown` for more than 30 s future skew, and no + observation when liveness or the activity stamp is absent; +5. prefer the session observation over a missing or derived-`unknown` driver + observation. + +The output clock and thresholds belong to the consumer: PTY reports when it +last observed output and does not interpret harness semantics. st2 neither +imports nor names a launcher. The session projection does not write +`harness-state`; it therefore has no writer identity, fencing, heartbeat, or +transport lifecycle. Its `blockedOn`, `ask`, and `inputBuffer` values remain +`unknown` because PTY output cannot prove them. + +`fidelity` is the discriminator that makes this partial tuple explicit: + +- `driver` — every axis follows the envelope semantics in this spec; +- `session` — only `state` and `since` are proved; consumers use those two + fields and must not poison coarse activity with the unknown fine axes. + ## Exposure (OHS-R09, OHS-R10) `st2 agents --json` (both forms) appends one field per row: ```json "observedState": { + "fidelity": "driver", "state": "active", "blockedOn": "human", "inputBuffer": "unknown", @@ -381,18 +419,30 @@ observed. } ``` -`null` when no record exists. The derivation above is already applied — a -consumer never re-implements staleness. Roster reads pass the same-host -liveness probe only for agents whose resolved host is this host, resolving -the pty root exactly as the runner does (`PTY_ROOT`, else the catalog's own -pty root; the legacy `PTY_SESSION_DIR` is deliberately not honored — the -runner never uses it, and probing a directory st2-managed sessions never -touch turns provable deaths into indeterminate reads). `status`, -`desiredState`, and `lastActivity` keep their exact meanings; the three -full-string pinned assertions and the stable-roster invariant wording are -updated deliberately in the change that adds the field. Doctor prints an -advisory (not a failure) for an owned agent whose record is stale, -session-dead, or `ended` while desired state is `running`. +The session projection uses the same object with an explicit discriminator: + +```json +"observedState": { + "fidelity": "session", + "state": "idle", + "blockedOn": "unknown", + "inputBuffer": "unknown", + "ask": "unknown", + "harness": null, + "since": 1787690000000, + "reason": null, + "exit": null +} +``` + +`null` only when neither a usable driver record nor local PTY session activity +exists. The derivation and precedence above are already applied — a consumer +never re-implements freshness or liveness. Roster reads resolve the pty root +exactly as the runner does (`PTY_ROOT`, else the catalog's own pty root; the +legacy `PTY_SESSION_DIR` is deliberately not honored). `status`, +`desiredState`, and `lastActivity` keep their exact meanings; pinned wire +assertions change deliberately with the discriminator. Doctor names fidelity, +warns on indeterminacy or missing both sources, and remains advisory. ## Verification plan @@ -410,6 +460,14 @@ each only once a real test proves it (per `CLAUDE.md`): escalation. Proving tests live in `src/harness_state.rs` today (11 tests) plus the planned per-producer suites; the SIGKILL-mid-turn test (`ended`, not `active`) gates the teardown row. +- **Session activity composition** — deterministic fixtures prove fresh vs + older vs future-skewed output, missing liveness/output evidence, and + definite-driver-over-session / session-over-indeterminate-driver precedence. + PTY integration tests prove absent-before-output, debounced persist after + output, subsequent-stamp advancement, and exit carrying the final stamp. +- **Fleet cost** — the experiment records the rejected alternatives and direct + metadata-read baseline. A large-catalog benchmark gates the consumer PR: + composed roster reads must remain linear and avoid subprocesses. ## Open design questions diff --git a/docs/vrs/ontology.md b/docs/vrs/ontology.md index 436c1b03..b6fa9edc 100644 --- a/docs/vrs/ontology.md +++ b/docs/vrs/ontology.md @@ -119,13 +119,17 @@ Authority: [`reconcile::Session`](../../src/reconcile.rs#L16-L26) ### observed harness state -The driver-written record of what a harness is seen doing: activity -(`idle`/`active`/`child`/`ended`, with `unknown` derived and never written), -who it is blocked on — and, when blocked on a human, what kind of ask holds -it (`permission`/`question`/`review`) — and what its input buffer holds. The observed -counterpart of the declared axes: it is not [presence](#presence) (agent- -authored availability), not [session state](#session-state) (task-record -liveness), not R08's *declared activity status*, and not R09's *working +The composed observation of what a harness session is seen doing. Its +`fidelity` discriminator names the proof: + +- **driver fidelity** — a driver-written record carrying activity, who it is + blocked on (and the ask kind), and input-buffer state; +- **session fidelity** — a launcher-agnostic, read-time projection from the + PTY's last output, proving coarse `active | idle` activity only. + +Both are the observed counterpart of the declared axes. Neither is +[presence](#presence) (agent-authored availability), [session state](#session-state) +(task-record liveness), R08's *declared activity status*, or R09's *working state* (restored context). Authority: [`harness_state`](../../src/harness_state.rs); diff --git a/src/agents.rs b/src/agents.rs index b1031906..043876d4 100644 --- a/src/agents.rs +++ b/src/agents.rs @@ -4,14 +4,29 @@ use std::fs; use std::path::{Path, PathBuf}; -use std::time::UNIX_EPOCH; +use std::time::{SystemTime, UNIX_EPOCH}; -use serde::Serialize; +use serde::{Deserialize, Serialize}; use crate::message; use crate::status::{self, State}; use crate::{AgentSpec, Discovered, Resource, harness_state}; +/// A PTY that emitted output inside this window is running a turn. Maintained +/// harnesses stream tokens or redraw progress continuously while active and go +/// silent between turns, so one minute leaves wide margin on both sides. +/// +/// Deliberately not an alias of presence or driver-record freshness: this is a +/// read-time session-activity projection with its own evidence and semantics. +const SESSION_ACTIVE_WINDOW_MS: u64 = 60_000; +const SESSION_FUTURE_SKEW_MS: u64 = 30_000; + +#[derive(Deserialize)] +#[serde(rename_all = "camelCase")] +struct PtySessionMetadata { + last_output_at_ms: Option, +} + /// One roster row: everything `st2 agents [--enrich]` can report about an agent. #[derive(Debug, Clone)] pub struct AgentRow { @@ -57,13 +72,15 @@ pub fn roster_from_discovered( this_host: &str, ) -> Vec { let pty_root = probe_pty_root(catalog_root); + let now_ms = unix_ms_now(); let mut rows: Vec = found .specs .iter() .filter_map(|s| { let agent_dir = s.path.parent()?; + let bus_id = s.bus_id(this_host); Some(AgentRow { - identity: s.bus_id(this_host), + identity: bus_id.clone(), status: status::read_state(&status::status_path(agent_dir)), name: s.name.clone(), description: s.description.clone(), @@ -73,7 +90,7 @@ pub fn roster_from_discovered( resources: s.resources.clone(), last_activity_ms: newest_activity_ms(agent_dir), inbox: inbox_count(agent_dir), - observed: observed_state(s, agent_dir, &pty_root, this_host), + observed: observed_state_at(s, agent_dir, &pty_root, this_host, &bus_id, now_ms), }) }) .collect(); @@ -81,24 +98,101 @@ pub fn roster_from_discovered( rows } -/// Read the observed-harness-state record beside `status`. The session-liveness cross-check is -/// host-local by construction: it applies only to agents this host runs, and an unreadable -/// registry downgrades nothing. -fn observed_state( +/// Read the same composed observation exposed by the roster, for consumers +/// such as Doctor that already hold one discovered Agent Spec. +pub fn read_observed_state( + spec: &AgentSpec, + agent_dir: &Path, + pty_root: &Path, + this_host: &str, +) -> Option { + let bus_id = spec.bus_id(this_host); + observed_state_at(spec, agent_dir, pty_root, this_host, &bus_id, unix_ms_now()) +} + +/// Compose the rich driver record with launcher-agnostic PTY session activity. +/// A definite, fresh driver record wins. A missing or indeterminate driver +/// record falls back to session activity on the local host. Cross-host readers +/// have neither the remote PTY registry nor its activity stamp, so they retain +/// the replicated driver record unchanged. +fn observed_state_at( spec: &AgentSpec, agent_dir: &Path, pty_root: &Path, this_host: &str, + bus_id: &str, + now_ms: u64, ) -> Option { let path = harness_state::harness_state_path(agent_dir); - if spec.resolved_host(this_host) == this_host { - let probe = |session: &str| crate::ding::session_liveness_in(pty_root, session); - harness_state::read(&path, Some(&probe)) + if spec.resolved_host(this_host) != this_host { + return harness_state::read(&path, None); + } + + let probe = |session: &str| crate::ding::session_liveness_in(pty_root, session); + let driver = harness_state::read(&path, Some(&probe)); + let session = session_observation(pty_root, bus_id, now_ms); + compose_observations(driver, session) +} + +fn compose_observations( + driver: Option, + session: Option, +) -> Option { + if driver + .as_ref() + .is_some_and(|observed| observed.state != harness_state::Activity::Unknown) + { + driver } else { - harness_state::read(&path, None) + session.or(driver) } } +/// Read coarse activity from the canonical agent task's PTY session. The +/// runner pins that session id to the agent bus id (`reconcile`'s canonical +/// agent rule), so the mapping is computed and requires no launcher knowledge. +fn session_observation( + pty_root: &Path, + bus_id: &str, + now_ms: u64, +) -> Option { + if crate::ding::session_liveness_in(pty_root, bus_id) != harness_state::SessionLiveness::Alive { + return None; + } + let metadata: PtySessionMetadata = + serde_json::from_slice(&fs::read(pty_root.join(format!("{bus_id}.json"))).ok()?).ok()?; + let last_output_at_ms = metadata.last_output_at_ms?; + let future_skew = last_output_at_ms.saturating_sub(now_ms); + let (state, reason) = if future_skew > SESSION_FUTURE_SKEW_MS { + ( + harness_state::Activity::Unknown, + Some("pty-future-skew".to_owned()), + ) + } else if now_ms.saturating_sub(last_output_at_ms) <= SESSION_ACTIVE_WINDOW_MS { + (harness_state::Activity::Active, None) + } else { + (harness_state::Activity::Idle, None) + }; + Some(harness_state::Observed { + fidelity: harness_state::Fidelity::Session, + state, + blocked_on: harness_state::BlockedOn::Unknown, + input_buffer: harness_state::InputBuffer::Unknown, + ask: harness_state::Ask::Unknown, + harness: None, + since_ms: Some(last_output_at_ms), + exit: None, + reason, + }) +} + +fn unix_ms_now() -> u64 { + SystemTime::now() + .duration_since(UNIX_EPOCH) + .map(|duration| duration.as_millis().min(u128::from(u64::MAX)) as u64) + .unwrap_or_default() +} + /// The pty registry root the probe reads: exactly the runner's own resolution, so the reader and /// the sessions it probes can never disagree. The runner honors `PTY_ROOT` and nothing else — a /// legacy `PTY_SESSION_DIR` here would point the probe at a directory st2-managed sessions never @@ -112,6 +206,7 @@ pub fn probe_pty_root(catalog_root: &Path) -> PathBuf { #[derive(Serialize)] #[serde(rename_all = "camelCase")] struct ObservedJson<'a> { + fidelity: &'a str, state: &'a str, blocked_on: &'a str, input_buffer: &'a str, @@ -125,6 +220,7 @@ struct ObservedJson<'a> { impl<'a> ObservedJson<'a> { fn from_row(observed: Option<&'a harness_state::Observed>) -> Option { observed.map(|observed| ObservedJson { + fidelity: observed.fidelity.as_str(), state: observed.state.as_str(), blocked_on: observed.blocked_on.as_str(), input_buffer: observed.input_buffer.as_str(), @@ -334,6 +430,7 @@ mod tests { 0, ); wedged.observed = Some(harness_state::Observed { + fidelity: harness_state::Fidelity::Driver, state: harness_state::Activity::Idle, blocked_on: harness_state::BlockedOn::None, input_buffer: harness_state::InputBuffer::Empty, @@ -346,15 +443,16 @@ mod tests { assert_eq!( to_json(&[wedged.clone()], false), - r#"[{"identity":"hetz.worker","status":"busy","name":null,"description":null,"retired":false,"resources":[],"desiredState":"running","desiredStateReason":null,"observedState":{"state":"idle","blockedOn":"none","inputBuffer":"empty","ask":"none","harness":"codex","since":1784653000000,"reason":null,"exit":null}}]"# + r#"[{"identity":"hetz.worker","status":"busy","name":null,"description":null,"retired":false,"resources":[],"desiredState":"running","desiredStateReason":null,"observedState":{"fidelity":"driver","state":"idle","blockedOn":"none","inputBuffer":"empty","ask":"none","harness":"codex","since":1784653000000,"reason":null,"exit":null}}]"# ); assert_eq!( to_json(&[wedged], true), - r#"[{"identity":"hetz.worker","status":"busy","name":null,"description":null,"retired":false,"resources":[],"lastActivity":1784653027733.6138,"inbox":0,"desiredState":"running","desiredStateReason":null,"observedState":{"state":"idle","blockedOn":"none","inputBuffer":"empty","ask":"none","harness":"codex","since":1784653000000,"reason":null,"exit":null}}]"# + r#"[{"identity":"hetz.worker","status":"busy","name":null,"description":null,"retired":false,"resources":[],"lastActivity":1784653027733.6138,"inbox":0,"desiredState":"running","desiredStateReason":null,"observedState":{"fidelity":"driver","state":"idle","blockedOn":"none","inputBuffer":"empty","ask":"none","harness":"codex","since":1784653000000,"reason":null,"exit":null}}]"# ); let mut derived = row("hetz.worker", State::Available, None, false, None, 0); derived.observed = Some(harness_state::Observed { + fidelity: harness_state::Fidelity::Driver, state: harness_state::Activity::Unknown, blocked_on: harness_state::BlockedOn::Unknown, input_buffer: harness_state::InputBuffer::Unknown, @@ -366,7 +464,115 @@ mod tests { }); assert_eq!( to_json(&[derived], false), - r#"[{"identity":"hetz.worker","status":"available","name":null,"description":null,"retired":false,"resources":[],"desiredState":"running","desiredStateReason":null,"observedState":{"state":"unknown","blockedOn":"unknown","inputBuffer":"unknown","ask":"unknown","harness":"codex","since":null,"reason":"session-dead","exit":null}}]"# + r#"[{"identity":"hetz.worker","status":"available","name":null,"description":null,"retired":false,"resources":[],"desiredState":"running","desiredStateReason":null,"observedState":{"fidelity":"driver","state":"unknown","blockedOn":"unknown","inputBuffer":"unknown","ask":"unknown","harness":"codex","since":null,"reason":"session-dead","exit":null}}]"# + ); + } + + fn write_pty_session(root: &Path, bus_id: &str, last_output_at_ms: Option) { + fs::write( + root.join(format!("{bus_id}.pid")), + std::process::id().to_string(), + ) + .unwrap(); + let metadata = match last_output_at_ms { + Some(timestamp) => serde_json::json!({ "lastOutputAtMs": timestamp }), + None => serde_json::json!({}), + }; + fs::write( + root.join(format!("{bus_id}.json")), + serde_json::to_vec(&metadata).unwrap(), + ) + .unwrap(); + } + + #[test] + fn session_activity_projects_active_idle_and_future_skew() { + let root = tempfile::tempdir().unwrap(); + let bus_id = "dev3.worker"; + let now = 1_800_000_000_000; + + write_pty_session(root.path(), bus_id, Some(now - 500)); + let active = session_observation(root.path(), bus_id, now).unwrap(); + assert_eq!(active.fidelity, harness_state::Fidelity::Session); + assert_eq!(active.state, harness_state::Activity::Active); + assert_eq!(active.blocked_on, harness_state::BlockedOn::Unknown); + assert_eq!(active.since_ms, Some(now - 500)); + + write_pty_session( + root.path(), + bus_id, + Some(now - SESSION_ACTIVE_WINDOW_MS - 1), + ); + let idle = session_observation(root.path(), bus_id, now).unwrap(); + assert_eq!(idle.state, harness_state::Activity::Idle); + assert_eq!(idle.reason, None); + + write_pty_session(root.path(), bus_id, Some(now + SESSION_FUTURE_SKEW_MS + 1)); + let skewed = session_observation(root.path(), bus_id, now).unwrap(); + assert_eq!(skewed.state, harness_state::Activity::Unknown); + assert_eq!(skewed.reason.as_deref(), Some("pty-future-skew")); + } + + #[test] + fn session_activity_requires_both_liveness_and_an_output_stamp() { + let root = tempfile::tempdir().unwrap(); + let bus_id = "dev3.worker"; + write_pty_session(root.path(), bus_id, None); + assert_eq!(session_observation(root.path(), bus_id, 10_000), None); + + fs::write(root.path().join(format!("{bus_id}.pid")), "0").unwrap(); + fs::write( + root.path().join(format!("{bus_id}.json")), + r#"{"lastOutputAtMs":9999}"#, + ) + .unwrap(); + assert_eq!(session_observation(root.path(), bus_id, 10_000), None); + } + + #[test] + fn fresh_driver_state_wins_and_indeterminate_driver_falls_back_to_session() { + let driver = harness_state::Observed { + fidelity: harness_state::Fidelity::Driver, + state: harness_state::Activity::Idle, + blocked_on: harness_state::BlockedOn::None, + input_buffer: harness_state::InputBuffer::Empty, + ask: harness_state::Ask::None, + harness: Some("codex".to_owned()), + since_ms: Some(10), + exit: None, + reason: None, + }; + let session = harness_state::Observed { + fidelity: harness_state::Fidelity::Session, + state: harness_state::Activity::Active, + blocked_on: harness_state::BlockedOn::Unknown, + input_buffer: harness_state::InputBuffer::Unknown, + ask: harness_state::Ask::Unknown, + harness: None, + since_ms: Some(20), + exit: None, + reason: None, + }; + + assert_eq!( + compose_observations(Some(driver.clone()), Some(session.clone())), + Some(driver) + ); + + let indeterminate = harness_state::Observed { + fidelity: harness_state::Fidelity::Driver, + state: harness_state::Activity::Unknown, + blocked_on: harness_state::BlockedOn::Unknown, + input_buffer: harness_state::InputBuffer::Unknown, + ask: harness_state::Ask::Unknown, + harness: Some("codex".to_owned()), + since_ms: None, + exit: None, + reason: Some("stale".to_owned()), + }; + assert_eq!( + compose_observations(Some(indeterminate), Some(session.clone())), + Some(session) ); } } diff --git a/src/harness_state.rs b/src/harness_state.rs index 57308839..95788997 100644 --- a/src/harness_state.rs +++ b/src/harness_state.rs @@ -533,11 +533,32 @@ impl Writer { } } +/// How much of the harness-state vocabulary the observation source can prove. +/// +/// Driver observations cover every axis in the envelope. Session observations +/// cover only coarse activity from PTY output; their blocked/input/ask axes stay +/// `unknown` and consumers must use only `state`. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Fidelity { + Driver, + Session, +} + +impl Fidelity { + pub fn as_str(self) -> &'static str { + match self { + Fidelity::Driver => "driver", + Fidelity::Session => "session", + } + } +} + /// The derived view a consumer reads. `state` already folds in staleness, future skew, /// malformation, and (when a probe is supplied) session liveness; `reason` names which derivation /// produced an `unknown`, so no absence is silent. #[derive(Debug, Clone, PartialEq)] pub struct Observed { + pub fidelity: Fidelity, pub state: Activity, pub blocked_on: BlockedOn, pub input_buffer: InputBuffer, @@ -553,6 +574,7 @@ impl Observed { // The single constructor for an indeterminate observation: every absence routes here, so // no path can derive `idle` — or anything else — from missing evidence. Self { + fidelity: Fidelity::Driver, state: Activity::Unknown, blocked_on: BlockedOn::Unknown, input_buffer: InputBuffer::Unknown, @@ -641,6 +663,7 @@ fn read_raw_at( } } Observed { + fidelity: Fidelity::Driver, state: record.state, blocked_on: record.blocked_on, input_buffer: record.input_buffer, diff --git a/src/main.rs b/src/main.rs index 28026fbd..aec658aa 100644 --- a/src/main.rs +++ b/src/main.rs @@ -1880,21 +1880,21 @@ fn doctor_cmd(root: &Path, host: Option, require_supervisor: bool) -> Re "rotted to `unknown` — is its session owner refreshing presence?", ); } - // Observed harness state is advisory-only in doctor: absence names a driver gap and - // a derived `unknown` names its reason, but neither fails the exit code. - let observed_path = st2::harness_state::harness_state_path(dir); + // Observed state is advisory-only in Doctor. A fresh driver record + // wins; otherwise the local PTY activity stamp supplies coarse + // session fidelity. Neither absence nor indeterminacy fails Doctor. let pty_root = st2::agents::probe_pty_root(&catalog); - let probe = |session: &str| st2::ding::session_liveness_in(&pty_root, session); - match st2::harness_state::read(&observed_path, Some(&probe)) { + match st2::agents::read_observed_state(spec, dir, &pty_root, &this_host) { None => report_advisory( - &format!("{bus_id} observed harness state absent"), - "no driver has published a harness-state record for this agent", + &format!("{bus_id} observed state absent"), + "no fresh driver record or PTY lastOutputAtMs stamp — upgrade/restart pty or inspect the session", ), Some(observed) if observed.state == st2::harness_state::Activity::Unknown => { report_advisory( - &format!("{bus_id} observed harness state indeterminate"), + &format!("{bus_id} observed state indeterminate"), &format!( - "derived `unknown` ({}) — is its driver still observing the harness?", + "{}-fidelity `unknown` ({})", + observed.fidelity.as_str(), observed.reason.as_deref().unwrap_or("unstated") ), ) @@ -1911,9 +1911,6 @@ fn doctor_cmd(root: &Path, host: Option, require_supervisor: bool) -> Re .exit .as_deref() .map(|exit| format!("exit {exit}")) - // A terminal record can carry only a reason — Codex's - // observed systemError writes reason without an exit — and - // discarding it leaves the operator nothing to act on. .or_else(|| { observed.reason.as_deref().map(|reason| format!("{reason}")) }) @@ -1925,7 +1922,8 @@ fn doctor_cmd(root: &Path, host: Option, require_supervisor: bool) -> Re &mut problems, true, &format!( - "{bus_id} observed harness state fresh (is `{}`)", + "{bus_id} observed state fresh ({} fidelity is `{}`)", + observed.fidelity.as_str(), observed.state.as_str() ), "", diff --git a/tests/doctor.rs b/tests/doctor.rs index 9357f5ce..a6e72aa8 100644 --- a/tests/doctor.rs +++ b/tests/doctor.rs @@ -470,7 +470,7 @@ fn observed_harness_state_arms_are_advisory_except_a_fresh_live_record() { let stdout = String::from_utf8_lossy(&absent.stdout); assert!(absent.status.success(), "{stdout}"); assert!( - stdout.contains("⚠ h.worker observed harness state absent"), + stdout.contains("⚠ h.worker observed state absent"), "{stdout}" ); @@ -492,7 +492,7 @@ fn observed_harness_state_arms_are_advisory_except_a_fresh_live_record() { let stdout = String::from_utf8_lossy(&live.stdout); assert!(live.status.success(), "{stdout}"); assert!( - stdout.contains("h.worker observed harness state fresh (is `active`)"), + stdout.contains("h.worker observed state fresh (driver fidelity is `active`)"), "{stdout}" ); @@ -505,7 +505,7 @@ fn observed_harness_state_arms_are_advisory_except_a_fresh_live_record() { stdout.contains("⚠ h.worker observed harness state ended"), "{stdout}" ); - assert!(stdout.contains("session ended (signal 9)"), "{stdout}"); + assert!(stdout.contains("session ended (exit signal 9)"), "{stdout}"); // A record that derives `unknown` names its reason, still advisory. fs::write(agent_dir.join("harness-state"), "garbage").unwrap(); @@ -513,7 +513,7 @@ fn observed_harness_state_arms_are_advisory_except_a_fresh_live_record() { let stdout = String::from_utf8_lossy(&indeterminate.stdout); assert!(indeterminate.status.success(), "{stdout}"); assert!( - stdout.contains("⚠ h.worker observed harness state indeterminate"), + stdout.contains("⚠ h.worker observed state indeterminate"), "{stdout}" ); assert!(stdout.contains("(malformed-record)"), "{stdout}"); From 047edc31baf541360e85df4dce9d76c814124b88 Mon Sep 17 00:00:00 2001 From: schickling-assistant <261620128+schickling-assistant@users.noreply.github.com> Date: Wed, 26 Aug 2026 20:25:23 +0200 Subject: [PATCH 2/2] docs(vrs): bind session activity to shared PTY wire agent-identity: unknown agent-persona: generalist agent-supervisor: unavailable agent-tool: OMP agent-tool-version: 18.0.3 agent-runtime: OMP 18.0.3 tooling-profile: dotfiles@e4789b0 --- docs/vrs/05-harness-state/spec.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/vrs/05-harness-state/spec.md b/docs/vrs/05-harness-state/spec.md index 5d828d72..076a89d3 100644 --- a/docs/vrs/05-harness-state/spec.md +++ b/docs/vrs/05-harness-state/spec.md @@ -376,6 +376,10 @@ feeding the same chunk to the terminal emulator. A trailing-edge one-second debounce persists the newest value as `lastOutputAtMs` through PTY's locked metadata mutation. Exit metadata carries the final in-memory stamp. +The Node and Rust PTY implementations expose the identical optional camelCase +metadata field and persistence semantics. st2 consumes that shared registry +contract, not an implementation-specific API. + Roster reads on the local host: 1. read and derive the driver record with its existing liveness probe;