diff --git a/INVARIANTS.md b/INVARIANTS.md index bd053af9..a12f3c32 100644 --- a/INVARIANTS.md +++ b/INVARIANTS.md @@ -11,7 +11,7 @@ materialization, messaging, DING, or presence must preserve them. | **Transport-decoupled lifecycle** | Each task is isolated from a supervisor/transport process-group or cgroup cascade. | `tests/transport_isolation.rs`; `tests/transport_isolation_macos.rs` | | **Clean exec teardown** | Killing an exec task reaps its whole process group. | `tests/exec_backend.rs::exec_kill_reaps_the_whole_process_group_not_just_the_leader` | | **Bounded restart diagnostics** | Relaunching an exec task preserves the just-finished log as one prior generation while bounding retained diagnostics to current plus prior. Final retirement removes the PID and both logs. | `tests/exec_backend.rs::exec_restart_reap_keeps_bounded_diagnostics_and_final_remove_cleans_them`; `tests/run.rs::up_once_finally_removes_dead_retired_tasks_without_restarting_them` | -| **Derived companion lifecycle** | A generated DING or launched stream starts only with an eligible canonical agent, is suppressed or stopped while that target is held, suspended, unavailable, retired, or terminally parked, and remains coupled without changing explicit sibling task behavior. A stream lowers its authored adapter launch directly, parks and surfaces independently, and neither makes an otherwise empty agent runnable nor claims a delivery transport. Stream authoring is serialized, authority-scoped, source-preserving, and fail-closed for Nix ownership and invalid declarations. Healthy compact startup launches companions in one pass, and targeted reconciliation holds a missing generated companion rather than broadening to its agent. | `crates/agent-spec/tests/discovery.rs::streams_are_typed_and_only_launched_streams_lower_to_derived_exec_tasks`; `crates/agent-spec/tests/discovery.rs::stream_names_launches_and_task_collisions_fail_closed`; `src/agent_author.rs::stream_add_supports_external_command_and_argv_and_remove_is_idempotent`; `src/agent_author.rs::stream_authoring_enforces_authority_nix_ownership_and_canonical_validation`; `tests/run.rs::fresh_compact_agent_launches_with_its_derived_ding`; `tests/run.rs::held_adopt_only_compact_agent_stops_its_live_derived_ding`; `tests/run.rs::parked_compact_agent_stops_its_live_derived_ding`; `tests/run.rs::retired_compact_agent_stops_agent_and_derived_ding`; `tests/run.rs::selected_missing_derived_ding_is_held_without_broadening_to_its_agent`; `tests/run.rs::fresh_compact_agent_launches_with_its_derived_stream`; `tests/run.rs::retired_compact_agent_stops_agent_and_derived_stream`; `tests/run.rs::suspended_compact_agent_stops_its_derived_stream_without_touching_a_sibling`; `tests/run.rs::held_adopt_only_compact_agent_stops_its_live_derived_stream`; `tests/run.rs::a_crash_looping_stream_parks_and_surfaces_without_disturbing_its_agent`; `tests/run.rs::parked_compact_agent_stops_its_live_derived_stream`; `tests/run.rs::selected_missing_derived_stream_is_held_without_broadening_to_its_agent`; `tests/run.rs::a_stream_alone_does_not_make_an_agent_runnable`; `tests/run.rs::a_stream_does_not_claim_a_delivery_transport` | +| **Derived companion lifecycle** | A generated DING or launched stream starts only with an eligible canonical agent, is suppressed or stopped while that target is held, suspended, unavailable, retired, or terminally parked, and remains coupled without changing explicit sibling task behavior. A stream lowers its authored adapter launch directly, parks and surfaces independently, and neither makes an otherwise empty agent runnable nor claims a delivery transport. Stream authoring is serialized, authority-scoped, source-preserving, and fail-closed for Nix ownership and invalid declarations. Healthy compact startup launches companions in one pass, and targeted reconciliation holds a missing generated companion rather than broadening to its agent. | `crates/agent-spec/tests/discovery.rs::streams_are_typed_and_only_launched_streams_lower_to_derived_exec_tasks`; `crates/agent-spec/tests/discovery.rs::stream_names_launches_and_task_collisions_fail_closed`; `src/agent_author.rs::stream_add_supports_external_command_and_argv_and_external_remove_is_idempotent`; `src/agent_author.rs::stream_authoring_enforces_authority_nix_ownership_and_canonical_validation`; `tests/run.rs::fresh_compact_agent_launches_with_its_derived_ding`; `tests/run.rs::held_adopt_only_compact_agent_stops_its_live_derived_ding`; `tests/run.rs::parked_compact_agent_stops_its_live_derived_ding`; `tests/run.rs::retired_compact_agent_stops_agent_and_derived_ding`; `tests/run.rs::selected_missing_derived_ding_is_held_without_broadening_to_its_agent`; `tests/run.rs::fresh_compact_agent_launches_with_its_derived_stream`; `tests/run.rs::retired_compact_agent_stops_agent_and_derived_stream`; `tests/run.rs::suspended_compact_agent_stops_its_derived_stream_without_touching_a_sibling`; `tests/run.rs::held_adopt_only_compact_agent_stops_its_live_derived_stream`; `tests/run.rs::a_crash_looping_stream_parks_and_surfaces_without_disturbing_its_agent`; `tests/run.rs::parked_compact_agent_stops_its_live_derived_stream`; `tests/run.rs::selected_missing_derived_stream_is_held_without_broadening_to_its_agent`; `tests/run.rs::a_launched_stream_alone_is_rejected_before_reconciliation`; `tests/run.rs::a_stream_does_not_claim_a_delivery_transport` | | **Exactly-once-safe native bus** | Messages use stable `-.md` files. An archive filename is a durable receipt that shadows and cleans restored inbox replicas and makes repeated archive cleanup idempotent. | `src/message.rs::filename_grammar`; `src/message.rs::archive_receipt_suppresses_and_idempotently_cleans_a_restored_inbox_copy`; `tests/message.rs` | | **Idempotent service requests** | A declared non-agent service principal publishes one exact JSON request per caller-supplied idempotency key to a canonical Agent Spec inbox. Concurrent or crash-replayed publication reuses the reserved filename; conflicting key reuse fails. The typed reply routes to the principal's canonical inbox without an Agent Spec identity or orphan mailbox. | `tests/request_cli.rs::stable_request_key_atomically_deduplicates_one_canonical_agent_message`; `tests/request_cli.rs::concurrent_replays_publish_exactly_one_request`; `tests/request_cli.rs::typed_reply_routes_to_the_principal_and_status_is_a_tagged_json_union`; `tests/request_cli.rs::request_api_rejects_agent_impersonation_and_unknown_flat_principals` | | **Bounded idempotent stream ingress** | A running agent accepts events only for a declared stream. Within the retained 128-receipt ring, concurrent or crash-replayed `(stream, event-id)` publication reuses one canonical filename, conflicting content fails, and supersession archives only the matching keyed predecessor or the stream-wide head through ordinary archive semantics. State remains bounded and honestly treats an identity evicted from the ring as new without searching inbox or archive history. Events do not write the Sent ledger and DING marks them as stream work. | `tests/event_e2e.rs::stable_event_identity_publishes_exactly_one_canonical_message`; `tests/event_e2e.rs::concurrent_replays_publish_exactly_one_event`; `tests/event_e2e.rs::conflicting_reuse_and_undeclared_or_suspended_ingress_fail_closed`; `tests/event_e2e.rs::supersede_collapses_only_the_matching_key_and_preserves_archive_receipts`; `tests/event_e2e.rs::keyless_supersede_replaces_the_stream_wide_head`; `tests/event_e2e.rs::crash_replay_honors_an_archive_receipt_and_never_restores_the_inbox_copy`; `tests/event_e2e.rs::subject_frontmatter_injection_is_refused_before_any_write`; `tests/event_e2e.rs::stream_state_is_bounded_and_forgets_only_beyond_its_honest_horizon`; `tests/event_e2e.rs::event_emit_cli_returns_a_stable_json_receipt_and_ding_marks_the_record`; `tests/stream_authoring_cli.rs::a_direct_adapter_launch_executes_the_exact_event_cli_contract`; `tests/run.rs::suspend_and_resume_relaunch_the_agent_and_stream_together` | @@ -19,7 +19,7 @@ materialization, messaging, DING, or presence must preserve them. | **Mutation-only filesystem wakeups** | Supervisor and DING filesystem watchers ignore read/open access events and wake early only for create, modify, rename, or remove events. Their own catalog and inbox reads therefore cannot bypass the bounded timer cadence or form a Linux inotify CPU loop. | `src/watch.rs::only_mutations_wake_watch_loops`; `src/watch.rs::linux_reads_are_silent_but_real_mutations_wake`; `src/ding/mod.rs::idle_ding_does_not_spin_on_its_own_inbox_reads`; `src/run.rs::idle_supervisor_does_not_spin_on_its_own_catalog_reads` | | **Bounded DING PTY probe churn** | An unsafe or active composer retains its FIFO notice but deferred delivery retries use a bounded backoff, so each inbox poll cannot spawn another short-lived PTY probe. | `src/ding/mod.rs::deferred_delivery_backoff_bounds_short_lived_pty_attempts` | | **Agent-declared presence discipline** | The shipped bus contract requires agents to declare `busy` before executing work, use `available` only while yielding or ready, and reserve `dnd` for an explicit hold. Both native harnesses materialize that contract. Busy remains observable but does not suppress DING; fresh `dnd` is the only delivery gate. | `tests/native_only.rs::clean_path_executes_the_maintained_native_authoring_guide`; `src/ding/mod.rs::pending_delivery_ignores_busy_but_respects_fresh_dnd_archive_and_retry` | -| **Stable roster JSON** | `st2 agents --json [--enrich]` preserves field names, order, null handling, presence, typed desired state and rationale, the retirement compatibility projection, opaque declared Resource descriptors, origin-timed activity, and inbox counts. Presence remains independent from desired lifecycle. | `src/agents.rs::agents_json_has_stable_wire_shape`; `src/agents.rs::agents_json_preserves_opaque_declared_resource_descriptors`; `tests/status_agents.rs::roster_json_and_human_output_distinguish_retirement_from_presence`; `tests/status_agents.rs::roster_keeps_presence_separate_from_suspended_desired_state`; `tests/status_agents.rs::roster_uses_version_1_origin_time_for_last_activity` | +| **Stable roster JSON** | `st2 agents --json [--enrich]` preserves field names, order, null handling, presence, typed desired state and rationale, the retirement compatibility projection, opaque declared Resource descriptors, origin-timed activity, inbox counts, and the appended `observedState` object. Declared presence, desired lifecycle, and observed harness state are three independent axes: none is derived from another, and a missing observed record is `null`, never a derived `unknown`. | `src/agents.rs::agents_json_has_stable_wire_shape`; `src/agents.rs::agents_json_preserves_opaque_declared_resource_descriptors`; `src/agents.rs::observed_state_joins_declared_presence_without_touching_either`; `tests/status_agents.rs::roster_json_and_human_output_distinguish_retirement_from_presence`; `tests/status_agents.rs::roster_keeps_presence_separate_from_suspended_desired_state`; `tests/status_agents.rs::roster_uses_version_1_origin_time_for_last_activity` | | **Agent-declared presence** | Refresh preserves non-DND declared status and advances the version 1 heartbeat. A missing status starts as `available`. Legacy DND migrates without renewing its hold. Version 1 DND is not refreshed. Stale, malformed, or implausibly future heartbeats read as `unknown`. The outer Codex, Claude, and pi session wrappers own a five-minute heartbeat while their provider remains alive. | `src/status.rs::refresh_preserves_value_and_changes_heartbeat_bytes`; `src/status.rs::refresh_upgrades_legacy_dnd_without_renewing_the_hold`; `src/status.rs::refresh_missing_writes_available_default`; `src/status.rs::version_1_staleness_and_future_skew_are_bounded`; `src/status.rs::malformed_versioned_record_is_unknown_without_mtime_fallback`; `src/claude_session.rs::idle_provider_refreshes_presence_without_mcp_input`; `src/pi_session.rs::idle_pi_provider_refreshes_presence_without_channel_input`; `src/codex_app_server.rs::inbox_fallback_does_not_write_a_fifteen_second_presence_heartbeat` | | **Retirement health** | A retired declaration is healthy only after every declared task ID is absent. Any live or dead declared task record reports incomplete retirement; retired declarations do not require presence. Live declarations retain their existing task and presence checks. | `tests/doctor.rs::retired_declaration_is_healthy_when_tasks_and_presence_are_absent`; `tests/doctor.rs::retired_declaration_is_unhealthy_while_a_declared_task_is_alive`; `tests/doctor.rs::retired_declaration_is_unhealthy_while_a_dead_task_record_remains` | | **Suspension health** | A suspended declaration is healthy when no declared task is live and every retained dead record is explicitly keep-pinned. It requires no presence, but this weaker result never proves retirement. Resume preserves ordinary keep and adopt-only policy. | `tests/doctor.rs::suspended_declaration_is_healthy_when_tasks_are_absent_without_presence`; `tests/doctor.rs::suspended_declaration_distinguishes_live_dead_keep_and_dead_nonkeep`; `tests/reconcile.rs::resuming_uses_ordinary_reconcile_and_does_not_override_keep` | diff --git a/src/agents.rs b/src/agents.rs index a2424b46..b1031906 100644 --- a/src/agents.rs +++ b/src/agents.rs @@ -10,7 +10,7 @@ use serde::Serialize; use crate::message; use crate::status::{self, State}; -use crate::{Discovered, Resource}; +use crate::{AgentSpec, Discovered, Resource, harness_state}; /// One roster row: everything `st2 agents [--enrich]` can report about an agent. #[derive(Debug, Clone)] @@ -36,18 +36,27 @@ pub struct AgentRow { pub last_activity_ms: Option, /// Count of canonical message files in the agent's inbox. `--enrich` only. pub inbox: usize, + /// Observed harness state — the driver-owned signal of what the harness is seen doing, a third + /// axis independent from declared presence and from desired lifecycle. `None` means no driver + /// has ever published a record for this agent, which is different from a derived `unknown`. + pub observed: Option, } /// Every agent in the catalog, sorted by bus id, with presence + enrich data computed. Read-only: /// walks discovered specs and each agent's resources, mutating nothing. pub fn roster(catalog_root: &Path, this_host: &str) -> Vec { let found = crate::discover(catalog_root); - roster_from_discovered(&found, this_host) + roster_from_discovered(&found, catalog_root, this_host) } /// Project a roster from one immutable discovery result. Exact selectors use this after proving /// discovery complete so the uniqueness check and returned metadata describe the same snapshot. -pub fn roster_from_discovered(found: &Discovered, this_host: &str) -> Vec { +pub fn roster_from_discovered( + found: &Discovered, + catalog_root: &Path, + this_host: &str, +) -> Vec { + let pty_root = probe_pty_root(catalog_root); let mut rows: Vec = found .specs .iter() @@ -64,6 +73,7 @@ pub fn roster_from_discovered(found: &Discovered, this_host: &str) -> Vec Vec Option { + let path = harness_state::harness_state_path(agent_dir); + if spec.resolved_host(this_host) == this_host { + let probe = |session: &str| crate::ding::session_liveness_in(pty_root, session); + harness_state::read(&path, Some(&probe)) + } else { + harness_state::read(&path, None) + } +} + +/// The pty registry root the probe reads: exactly the runner's own resolution, so the reader and +/// the sessions it probes can never disagree. The runner honors `PTY_ROOT` and nothing else — a +/// legacy `PTY_SESSION_DIR` here would point the probe at a directory st2-managed sessions never +/// use, turning provable deaths into indeterminate reads. +pub fn probe_pty_root(catalog_root: &Path) -> PathBuf { + crate::run::effective_pty_root(catalog_root) +} + +/// The `observedState` object inside a roster row. Vocabulary words are the record's own +/// (`as_str`), never Rust identifier spellings. +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct ObservedJson<'a> { + state: &'a str, + blocked_on: &'a str, + input_buffer: &'a str, + ask: &'a str, + harness: Option<&'a str>, + since: Option, + reason: Option<&'a str>, + exit: Option<&'a str>, +} + +impl<'a> ObservedJson<'a> { + fn from_row(observed: Option<&'a harness_state::Observed>) -> Option { + observed.map(|observed| ObservedJson { + state: observed.state.as_str(), + blocked_on: observed.blocked_on.as_str(), + input_buffer: observed.input_buffer.as_str(), + ask: observed.ask.as_str(), + harness: observed.harness.as_deref(), + since: observed.since_ms, + reason: observed.reason.as_deref(), + exit: observed.exit.as_deref(), + }) + } +} + /// `st2 agents --json` row. Field order and names are the stable wire contract. #[derive(Serialize)] struct SummaryJson<'a> { @@ -84,6 +150,8 @@ struct SummaryJson<'a> { desired_state: &'a str, #[serde(rename = "desiredStateReason")] desired_state_reason: Option<&'a str>, + #[serde(rename = "observedState")] + observed_state: Option>, } /// `st2 agents --json --enrich` row (adds `lastActivity` and `inbox`). @@ -102,6 +170,8 @@ struct EnrichedJson<'a> { desired_state: &'a str, #[serde(rename = "desiredStateReason")] desired_state_reason: Option<&'a str>, + #[serde(rename = "observedState")] + observed_state: Option>, } /// Serialize a roster to the stable JSON emitted by `st2 agents --json [--enrich]`. @@ -120,6 +190,7 @@ pub fn to_json(rows: &[AgentRow], enrich: bool) -> String { desired_state_reason: r.desired_state_reason.as_deref(), last_activity: r.last_activity_ms, inbox: r.inbox, + observed_state: ObservedJson::from_row(r.observed.as_ref()), }) .collect(); serde_json::to_string(&out).unwrap_or_else(|_| "[]".to_string()) @@ -135,6 +206,7 @@ pub fn to_json(rows: &[AgentRow], enrich: bool) -> String { resources: &r.resources, desired_state: &r.desired_state, desired_state_reason: r.desired_state_reason.as_deref(), + observed_state: ObservedJson::from_row(r.observed.as_ref()), }) .collect(); serde_json::to_string(&out).unwrap_or_else(|_| "[]".to_string()) @@ -199,6 +271,7 @@ mod tests { resources: Vec::new(), last_activity_ms: last, inbox, + observed: None, } } @@ -219,11 +292,11 @@ mod tests { assert_eq!( to_json(&rows, false), - r#"[{"identity":"hetz.cos-claude","status":"available","name":null,"description":null,"retired":false,"resources":[],"desiredState":"running","desiredStateReason":null},{"identity":"hetz.st2-claude","status":"busy","name":"owner","description":null,"retired":true,"resources":[],"desiredState":"retired","desiredStateReason":null}]"# + r#"[{"identity":"hetz.cos-claude","status":"available","name":null,"description":null,"retired":false,"resources":[],"desiredState":"running","desiredStateReason":null,"observedState":null},{"identity":"hetz.st2-claude","status":"busy","name":"owner","description":null,"retired":true,"resources":[],"desiredState":"retired","desiredStateReason":null,"observedState":null}]"# ); assert_eq!( to_json(&rows, true), - r#"[{"identity":"hetz.cos-claude","status":"available","name":null,"description":null,"retired":false,"resources":[],"lastActivity":1784653027733.6138,"inbox":1,"desiredState":"running","desiredStateReason":null},{"identity":"hetz.st2-claude","status":"busy","name":"owner","description":null,"retired":true,"resources":[],"lastActivity":null,"inbox":0,"desiredState":"retired","desiredStateReason":null}]"# + r#"[{"identity":"hetz.cos-claude","status":"available","name":null,"description":null,"retired":false,"resources":[],"lastActivity":1784653027733.6138,"inbox":1,"desiredState":"running","desiredStateReason":null,"observedState":null},{"identity":"hetz.st2-claude","status":"busy","name":"owner","description":null,"retired":true,"resources":[],"lastActivity":null,"inbox":0,"desiredState":"retired","desiredStateReason":null,"observedState":null}]"# ); // Empty roster is `[]`, not `null`. assert_eq!(to_json(&[], true), "[]"); @@ -243,7 +316,57 @@ mod tests { assert_eq!( to_json(&[resource_row], false), - r#"[{"identity":"hetz.worker","status":"available","name":null,"description":null,"retired":false,"resources":[{"name":"work","uri":"vendor+thing://authority/exact%20identity","reason":"Current implementation task."}],"desiredState":"running","desiredStateReason":null}]"# + r#"[{"identity":"hetz.worker","status":"available","name":null,"description":null,"retired":false,"resources":[{"name":"work","uri":"vendor+thing://authority/exact%20identity","reason":"Current implementation task."}],"desiredState":"running","desiredStateReason":null,"observedState":null}]"# + ); + } + + /// Declared presence and observed harness state are independent axes in one payload: a + /// declared `busy` sits beside an observed `idle` (the wedged-agent signal), and + /// `lastActivity` keeps its existing meaning untouched by the new field. + #[test] + fn observed_state_joins_declared_presence_without_touching_either() { + let mut wedged = row( + "hetz.worker", + State::Busy, + None, + false, + Some(1784653027733.6138), + 0, + ); + wedged.observed = Some(harness_state::Observed { + state: harness_state::Activity::Idle, + blocked_on: harness_state::BlockedOn::None, + input_buffer: harness_state::InputBuffer::Empty, + ask: harness_state::Ask::None, + harness: Some("codex".to_string()), + since_ms: Some(1784653000000), + exit: None, + reason: None, + }); + + assert_eq!( + to_json(&[wedged.clone()], false), + r#"[{"identity":"hetz.worker","status":"busy","name":null,"description":null,"retired":false,"resources":[],"desiredState":"running","desiredStateReason":null,"observedState":{"state":"idle","blockedOn":"none","inputBuffer":"empty","ask":"none","harness":"codex","since":1784653000000,"reason":null,"exit":null}}]"# + ); + assert_eq!( + to_json(&[wedged], true), + r#"[{"identity":"hetz.worker","status":"busy","name":null,"description":null,"retired":false,"resources":[],"lastActivity":1784653027733.6138,"inbox":0,"desiredState":"running","desiredStateReason":null,"observedState":{"state":"idle","blockedOn":"none","inputBuffer":"empty","ask":"none","harness":"codex","since":1784653000000,"reason":null,"exit":null}}]"# + ); + + let mut derived = row("hetz.worker", State::Available, None, false, None, 0); + derived.observed = Some(harness_state::Observed { + state: harness_state::Activity::Unknown, + blocked_on: harness_state::BlockedOn::Unknown, + input_buffer: harness_state::InputBuffer::Unknown, + ask: harness_state::Ask::Unknown, + harness: Some("codex".to_string()), + since_ms: None, + exit: None, + reason: Some("session-dead".to_string()), + }); + assert_eq!( + to_json(&[derived], false), + r#"[{"identity":"hetz.worker","status":"available","name":null,"description":null,"retired":false,"resources":[],"desiredState":"running","desiredStateReason":null,"observedState":{"state":"unknown","blockedOn":"unknown","inputBuffer":"unknown","ask":"unknown","harness":"codex","since":null,"reason":"session-dead","exit":null}}]"# ); } } diff --git a/src/ding/mod.rs b/src/ding/mod.rs index b7650104..e81f6a54 100644 --- a/src/ding/mod.rs +++ b/src/ding/mod.rs @@ -709,6 +709,44 @@ pub fn session_alive(session: &str) -> bool { pid > 0 && unsafe { libc::kill(pid, 0) == 0 } } +/// Positive-evidence session liveness for observed-harness-state readers. Unlike +/// [`session_alive`], whose delivery callers must fail closed ("any miss means gone"), a reader +/// deriving `unknown` needs proof of death: an unreadable or unparseable pidfile is +/// `Indeterminate` — the reader may not share the writer's PTY root — and a pid that exists but +/// is not signalable (EPERM) is still alive. +pub fn session_liveness(session: &str) -> crate::harness_state::SessionLiveness { + session_liveness_in(&pty_session_dir(), session) +} + +/// [`session_liveness`], probing an explicit registry root instead of the ambient environment — +/// readers that know the catalog derive the runner's own root rather than requiring PTY_ROOT in +/// the shell. +pub fn session_liveness_in( + root: &std::path::Path, + session: &str, +) -> crate::harness_state::SessionLiveness { + use crate::harness_state::SessionLiveness; + let pidfile = root.join(format!("{session}.pid")); + let Ok(raw) = std::fs::read_to_string(&pidfile) else { + return SessionLiveness::Indeterminate; + }; + let Ok(pid) = raw.trim().parse::() else { + return SessionLiveness::Indeterminate; + }; + if pid <= 0 { + return SessionLiveness::Indeterminate; + } + if unsafe { libc::kill(pid, 0) } == 0 { + return SessionLiveness::Alive; + } + match std::io::Error::last_os_error().raw_os_error() { + Some(code) if code == libc::ESRCH => SessionLiveness::Dead, + // EPERM proves existence; anything else proves nothing. + Some(code) if code == libc::EPERM => SessionLiveness::Alive, + _ => SessionLiveness::Indeterminate, + } +} + /// The `pty` session registry dir. This must mirror the sibling tool's resolution order. fn pty_session_dir() -> PathBuf { for var in ["PTY_ROOT", "PTY_SESSION_DIR"] { @@ -3751,4 +3789,42 @@ Enter to select · ↑/↓ to navigate · Esc to cancel"; "idle DING must sleep at its configured cadence" ); } + + #[test] + fn session_liveness_probe_reports_only_positive_evidence() { + use crate::harness_state::SessionLiveness; + let tmp = tempfile::tempdir().unwrap(); + + // An unreadable registry proves nothing: missing or garbled pidfiles downgrade nothing. + assert_eq!( + session_liveness_in(tmp.path(), "absent"), + SessionLiveness::Indeterminate + ); + std::fs::write(tmp.path().join("garbled.pid"), "not-a-pid\n").unwrap(); + assert_eq!( + session_liveness_in(tmp.path(), "garbled"), + SessionLiveness::Indeterminate + ); + + // Our own pid is positive evidence of life. + std::fs::write( + tmp.path().join("live.pid"), + format!("{}\n", std::process::id()), + ) + .unwrap(); + assert_eq!( + session_liveness_in(tmp.path(), "live"), + SessionLiveness::Alive + ); + + // A reaped child is positive evidence of death (ESRCH). + let mut child = std::process::Command::new("true").spawn().unwrap(); + let pid = child.id(); + child.wait().unwrap(); + std::fs::write(tmp.path().join("dead.pid"), format!("{pid}\n")).unwrap(); + assert_eq!( + session_liveness_in(tmp.path(), "dead"), + SessionLiveness::Dead + ); + } } diff --git a/src/main.rs b/src/main.rs index 94a618fd..11586395 100644 --- a/src/main.rs +++ b/src/main.rs @@ -1813,6 +1813,57 @@ fn doctor_cmd(root: &Path, host: Option, require_supervisor: bool) -> Re "rotted to `unknown` — is its session owner refreshing presence?", ); } + // Observed harness state is advisory-only in doctor: absence names a driver gap and + // a derived `unknown` names its reason, but neither fails the exit code. + let observed_path = st2::harness_state::harness_state_path(dir); + let pty_root = st2::agents::probe_pty_root(&catalog); + let probe = |session: &str| st2::ding::session_liveness_in(&pty_root, session); + match st2::harness_state::read(&observed_path, Some(&probe)) { + None => report_advisory( + &format!("{bus_id} observed harness state absent"), + "no driver has published a harness-state record for this agent", + ), + Some(observed) if observed.state == st2::harness_state::Activity::Unknown => { + report_advisory( + &format!("{bus_id} observed harness state indeterminate"), + &format!( + "derived `unknown` ({}) — is its driver still observing the harness?", + observed.reason.as_deref().unwrap_or("unstated") + ), + ) + } + Some(observed) + if observed.state == st2::harness_state::Activity::Ended + && spec.desired_state.as_str() == "running" => + { + report_advisory( + &format!("{bus_id} observed harness state ended"), + &format!( + "session ended ({}) while desired state is running — crashed seat?", + observed + .exit + .as_deref() + .map(|exit| format!("exit {exit}")) + // A terminal record can carry only a reason — Codex's + // observed systemError writes reason without an exit — and + // discarding it leaves the operator nothing to act on. + .or_else(|| { + observed.reason.as_deref().map(|reason| format!("{reason}")) + }) + .unwrap_or_else(|| "exit unstated".to_string()) + ), + ) + } + Some(observed) => report_check( + &mut problems, + true, + &format!( + "{bus_id} observed harness state fresh (is `{}`)", + observed.state.as_str() + ), + "", + ), + } } } @@ -2116,7 +2167,7 @@ fn agents_cmd( found.errors.len() ); } - let mut rows = st2::agents::roster_from_discovered(&found, &host); + let mut rows = st2::agents::roster_from_discovered(&found, &root, &host); if let Some(identity) = &identity { rows.retain(|row| row.identity == *identity); anyhow::ensure!( @@ -2151,9 +2202,10 @@ fn agents_cmd( ) }; println!( - "{}\t{}\t{}\t{}{}", + "{}\t{}\t{}\t{}\t{}{}", r.identity, r.status.as_str(), + observed_column(r.observed.as_ref()), r.name.as_deref().unwrap_or(""), r.description.as_deref().unwrap_or(""), lifecycle, @@ -2163,6 +2215,26 @@ fn agents_cmd( Ok(()) } +/// The compact observed-harness-state column for human `st2 agents` output. `-` means no record +/// exists (no driver has published one), which is distinct from a derived `unknown`. +fn observed_column(observed: Option<&st2::harness_state::Observed>) -> String { + // Prefixed so the observed word is never mistaken for the declared presence beside it — + // two bare state words in one row is the exact ambiguity the ontology's collision rules name. + let Some(observed) = observed else { + return "obs:-".to_string(); + }; + let mut column = observed.state.as_str().to_string(); + if observed.blocked_on == st2::harness_state::BlockedOn::Human { + column.push_str("+human"); + } + if observed.state == st2::harness_state::Activity::Unknown + && let Some(reason) = observed.reason.as_deref() + { + column = format!("unknown({reason})"); + } + format!("obs:{column}") +} + fn ding_cmd( session: Option, identity: Option, diff --git a/tests/doctor.rs b/tests/doctor.rs index b0da7b42..9357f5ce 100644 --- a/tests/doctor.rs +++ b/tests/doctor.rs @@ -442,3 +442,79 @@ fn missing_delivery_is_advisory_while_an_invalid_delivery_is_a_catalog_problem() "{stdout}" ); } + +#[test] +fn observed_harness_state_arms_are_advisory_except_a_fresh_live_record() { + use st2::harness_state::{Activity, BlockedOn, InputBuffer, Observation, Writer}; + + let tmp = tempfile::tempdir().unwrap(); + let catalog = tmp.path().join("catalog"); + let declaration = catalog.join("agents/h/worker/agent.kdl"); + let bin = tmp.path().join("bin"); + fs::create_dir_all(declaration.parent().unwrap()).unwrap(); + fs::create_dir_all(&bin).unwrap(); + fs::write( + &declaration, + r#"agent "worker" { host "h"; command "true"; deliver "mcp" }"#, + ) + .unwrap(); + let agent_dir = declaration.parent().unwrap().to_path_buf(); + fs::write(agent_dir.join("status"), "available\n").unwrap(); + executable( + &bin.join("pty"), + "#!/bin/sh\nif [ \"$1\" = list ]; then printf '[{\"name\":\"h.worker\",\"status\":\"running\"}]\\n'; fi\n", + ); + + // No record: a driver gap is an advisory, never a failing check. + let absent = doctor(&catalog, &bin, &tmp.path().join("state")); + let stdout = String::from_utf8_lossy(&absent.stdout); + assert!(absent.status.success(), "{stdout}"); + assert!( + stdout.contains("⚠ h.worker observed harness state absent"), + "{stdout}" + ); + + // A fresh live record is a passing check. + let mut writer = Writer::new( + &agent_dir, + "h.worker", + "codex", + Some("h.worker".to_string()), + ); + writer + .observe(Observation::new( + Activity::Active, + BlockedOn::None, + InputBuffer::Unknown, + )) + .unwrap(); + let live = doctor(&catalog, &bin, &tmp.path().join("state")); + let stdout = String::from_utf8_lossy(&live.stdout); + assert!(live.status.success(), "{stdout}"); + assert!( + stdout.contains("h.worker observed harness state fresh (is `active`)"), + "{stdout}" + ); + + // A terminal record while the declaration wants the seat running is the crashed-seat signal. + writer.ended("signal 9").unwrap(); + let ended = doctor(&catalog, &bin, &tmp.path().join("state")); + let stdout = String::from_utf8_lossy(&ended.stdout); + assert!(ended.status.success(), "{stdout}"); + assert!( + stdout.contains("⚠ h.worker observed harness state ended"), + "{stdout}" + ); + assert!(stdout.contains("session ended (signal 9)"), "{stdout}"); + + // A record that derives `unknown` names its reason, still advisory. + fs::write(agent_dir.join("harness-state"), "garbage").unwrap(); + let indeterminate = doctor(&catalog, &bin, &tmp.path().join("state")); + let stdout = String::from_utf8_lossy(&indeterminate.stdout); + assert!(indeterminate.status.success(), "{stdout}"); + assert!( + stdout.contains("⚠ h.worker observed harness state indeterminate"), + "{stdout}" + ); + assert!(stdout.contains("(malformed-record)"), "{stdout}"); +} diff --git a/tests/status_agents.rs b/tests/status_agents.rs index 44cb3220..66af0af0 100644 --- a/tests/status_agents.rs +++ b/tests/status_agents.rs @@ -286,7 +286,7 @@ fn roster_json_and_human_output_distinguish_retirement_from_presence() { ); assert_eq!( String::from_utf8(human.stdout).unwrap(), - "h.live\tavailable\t\t\nh.retired\tbusy\t\t\t[retired]\n" + "h.live\tavailable\tobs:-\t\t\nh.retired\tbusy\tobs:-\t\t\t[retired]\n" ); }