diff --git a/INVARIANTS.md b/INVARIANTS.md index a12f3c32..d0bc7b71 100644 --- a/INVARIANTS.md +++ b/INVARIANTS.md @@ -21,6 +21,8 @@ materialization, messaging, DING, or presence must preserve them. | **Agent-declared presence discipline** | The shipped bus contract requires agents to declare `busy` before executing work, use `available` only while yielding or ready, and reserve `dnd` for an explicit hold. Both native harnesses materialize that contract. Busy remains observable but does not suppress DING; fresh `dnd` is the only delivery gate. | `tests/native_only.rs::clean_path_executes_the_maintained_native_authoring_guide`; `src/ding/mod.rs::pending_delivery_ignores_busy_but_respects_fresh_dnd_archive_and_retry` | | **Stable roster JSON** | `st2 agents --json [--enrich]` preserves field names, order, null handling, presence, typed desired state and rationale, the retirement compatibility projection, opaque declared Resource descriptors, origin-timed activity, inbox counts, and the appended `observedState` object. Declared presence, desired lifecycle, and observed harness state are three independent axes: none is derived from another, and a missing observed record is `null`, never a derived `unknown`. | `src/agents.rs::agents_json_has_stable_wire_shape`; `src/agents.rs::agents_json_preserves_opaque_declared_resource_descriptors`; `src/agents.rs::observed_state_joins_declared_presence_without_touching_either`; `tests/status_agents.rs::roster_json_and_human_output_distinguish_retirement_from_presence`; `tests/status_agents.rs::roster_keeps_presence_separate_from_suspended_desired_state`; `tests/status_agents.rs::roster_uses_version_1_origin_time_for_last_activity` | | **Agent-declared presence** | Refresh preserves non-DND declared status and advances the version 1 heartbeat. A missing status starts as `available`. Legacy DND migrates without renewing its hold. Version 1 DND is not refreshed. Stale, malformed, or implausibly future heartbeats read as `unknown`. The outer Codex, Claude, and pi session wrappers own a five-minute heartbeat while their provider remains alive. | `src/status.rs::refresh_preserves_value_and_changes_heartbeat_bytes`; `src/status.rs::refresh_upgrades_legacy_dnd_without_renewing_the_hold`; `src/status.rs::refresh_missing_writes_available_default`; `src/status.rs::version_1_staleness_and_future_skew_are_bounded`; `src/status.rs::malformed_versioned_record_is_unknown_without_mtime_fallback`; `src/claude_session.rs::idle_provider_refreshes_presence_without_mcp_input`; `src/pi_session.rs::idle_pi_provider_refreshes_presence_without_channel_input`; `src/codex_app_server.rs::inbox_fallback_does_not_write_a_fifteen_second_presence_heartbeat` | +| **Scoped delivery-input wakeups** | Native delivery pumps watch only their inputs: the agent's `resources/inbox` subtree and its `status` file. Runtime records written beside them by the pump's own process group — presence temp siblings, the `harness-state` record, stream state — never wake delivery, so a producer that writes on every turn boundary cannot pump its own delivery loop. | `src/watch.rs::delivery_watcher_ignores_runtime_records_but_wakes_on_inbox_and_status` | +| **Observed harness state discipline** | The `harness-state` record is written only by the owning session's driver processes — wrapper, channel, or hooks — serialized by a cross-process lock and coalesced against the on-disk record, atomically and byte-distinct on every write that lands, with freshness from its embedded timestamp and never file mtime; restating an unchanged state touches the record only when the refresh cadence is due. `unknown` is derived, never written: staleness, future skew, malformation, an unsupported schema, and a provably dead pty session each read as `unknown` with a distinct reason; an indeterminate liveness probe downgrades nothing; a missing record is no observation rather than `unknown`; no absence derives a definite state. A writer that loses sight of its harness stops heartbeating instead of refreshing a state it cannot see, and a predecessor session's record is never re-stamped. A reaped provider yields a terminal `ended` record carrying its real exit — written before the stop path's SIGKILL escalation and rewritten from the escalation cover when a grace-window reap observes the real status, proven against the real wrapper binaries of both stop implementations — and never a live state. | `src/harness_state.rs::unknown_state_is_derived_and_cannot_be_written`; `src/harness_state.rs::malformed_record_is_unknown_without_mtime_fallback`; `src/harness_state.rs::staleness_and_future_skew_derive_unknown_with_distinct_reasons`; `src/harness_state.rs::a_dead_session_reads_unknown_even_while_fresh_but_ended_survives`; `src/harness_state.rs::every_landed_write_is_byte_distinct_and_fresh_restatements_do_not_write`; `src/harness_state.rs::a_chatty_producer_restating_its_state_causes_zero_writes`; `src/harness_state.rs::concurrent_writers_defer_to_the_on_disk_record_not_their_cache`; `src/harness_state.rs::a_predecessor_sessions_record_is_never_heartbeat_eligible`; `src/harness_state.rs::missing_record_reads_as_none_not_unknown`; `src/codex_app_server.rs::pump_publishes_observations_and_stops_heartbeating_on_evidence_loss`; `src/claude_session.rs::a_provider_killed_mid_turn_reads_ended_rather_than_active`; `src/claude_session.rs::a_clean_provider_exit_writes_the_terminal_record`; `tests/harness_state_teardown.rs::stop_escalation_writes_the_terminal_record_before_sigkill`; `tests/harness_state_teardown.rs::opencode_stop_escalation_writes_the_cover_record_before_sigkill`; `tests/harness_state_teardown.rs::opencode_graceful_stop_records_the_real_reaped_exit` | | **Retirement health** | A retired declaration is healthy only after every declared task ID is absent. Any live or dead declared task record reports incomplete retirement; retired declarations do not require presence. Live declarations retain their existing task and presence checks. | `tests/doctor.rs::retired_declaration_is_healthy_when_tasks_and_presence_are_absent`; `tests/doctor.rs::retired_declaration_is_unhealthy_while_a_declared_task_is_alive`; `tests/doctor.rs::retired_declaration_is_unhealthy_while_a_dead_task_record_remains` | | **Suspension health** | A suspended declaration is healthy when no declared task is live and every retained dead record is explicitly keep-pinned. It requires no presence, but this weaker result never proves retirement. Resume preserves ordinary keep and adopt-only policy. | `tests/doctor.rs::suspended_declaration_is_healthy_when_tasks_are_absent_without_presence`; `tests/doctor.rs::suspended_declaration_distinguishes_live_dead_keep_and_dead_nonkeep`; `tests/reconcile.rs::resuming_uses_ordinary_reconcile_and_does_not_override_keep` | | **Crash loops surface** | A task parked by a fail-mode restart policy notifies its supervisor once over the bus. | `tests/run.rs::surface_crash_loop_notifies_the_supervisor_over_the_bus` | diff --git a/docs/vrs/.delta/DELTA-005-harness-state-specified-ahead-of-full-implementation.md b/docs/vrs/.delta/DELTA-005-harness-state-specified-ahead-of-full-implementation.md deleted file mode 100644 index c2f1918e..00000000 --- a/docs/vrs/.delta/DELTA-005-harness-state-specified-ahead-of-full-implementation.md +++ /dev/null @@ -1,44 +0,0 @@ -# DELTA-005: observed harness state is specified ahead of its producers - -## Current mismatch - -[`05-harness-state`](../05-harness-state/spec.md) and decision -[`0006`](../.decisions/0006-observed-harness-state-is-a-driver-written-catalog-record.md) -specify one observed-state record with producers on all four maintained -harnesses, a roster join, Doctor exposure, and scoped delivery-input watching. -The shipped implementation at the time this delta was filed is the envelope -alone: `src/harness_state.rs` (record, writer, derivation, liveness -cross-check hook) with its module tests. No producer writes the record, the -roster does not expose it, the Codex delivery pump still watches its whole -agent directory unfiltered, and OpenCode has no typed driver. - -## Why the docs land first - -The vocabulary and derivation rules are the contract the fractal TUI pins; -they were interview-settled on 2026-08-23 and are cheaper to review as one -document set than re-derived per producer PR. Producers are independent -vertical slices (Codex projection, Claude hooks + wrapper, pi extension, -OpenCode driver) and land incrementally against this spec. - -## Direction - -Update implementation to match the spec, in dependency order. - -## Resolution Signal - -Checked off as each lands with its named tests green; this delta is deleted by -the change that completes the last box. - -- [ ] Scoped delivery-input watching replaces the Codex pump's unfiltered - agent-dir watch, with the no-self-wake regression test. -- [ ] Codex producer: projection per the spec table, wired to the control - pump's transitions, heartbeat, and terminal write. -- [ ] Claude producer: hook-side classification plus wrapper-owned heartbeat - and terminal write (pre-escalation ordering fixed). -- [ ] pi producer: evented extension transitions over the channel; wrapper - writes. -- [ ] OpenCode: typed driver + `opencode-session` wrapper (presence lease), - then its producer, then the native delivery transport. -- [ ] Roster `observedState` join with the pinned assertions updated and the - stable-roster invariant wording amended in the same change. -- [ ] Doctor advisory for owned agents. diff --git a/docs/vrs/05-harness-state/open-questions.md b/docs/vrs/05-harness-state/open-questions.md index 0bfa0009..907c73e8 100644 --- a/docs/vrs/05-harness-state/open-questions.md +++ b/docs/vrs/05-harness-state/open-questions.md @@ -25,13 +25,17 @@ hypotheses. waits on) and it under-reports nothing, but the state axis says `active` while the model is not running. Resolves by: a Claude build whose denial emits any hook event; until then the deny window is the pinned limit. -- **DQ-H2 Transport cost of per-transition writes.** Presence refreshes every - five minutes; turn boundaries are far more frequent, and burst coalescing - measured 4 transitions per turn 0.1–0.4 ms apart. No measurement establishes - what per-transition replicated writes cost on a real catalog under a real - transport (OHS-T01 accepts this for v1). Resolves by: measuring write and - sync volume on a live catalog; if unacceptable, a minimum-interval - coalescing window is the tuning knob, at the cost of spinner latency. +- **DQ-H2 Transport cost of per-transition writes — measured 2026-08-23, + resolved by the restatement guard.** The live smoke run caught the failure + mode: the OpenCode producer restated its state per SSE frame and the + envelope re-stamped every restatement — 679 byte-distinct writes in 221 s + (~2.7/s while idle). The envelope now makes an unchanged observation a + no-op until the refresh cadence is due, so a seat writes on transitions + plus at most one re-stamp per five minutes (a Claude turn measured 3 + writes; a pi turn 2–3). What remains open is only the fleet-scale sync + question: nothing yet measures what transition-rate writes cost a + 600-seat catalog's transport over a day. Resolves by: that measurement on + a live catalog. - **DQ-H3 `child` has no producer.** The word is reserved because the tuple's reasoning needs it (a long-running foreground command is neither the model working nor idle), but the producer that would have supplied it — the PTY diff --git a/docs/vrs/05-harness-state/spec.md b/docs/vrs/05-harness-state/spec.md index c59d9470..8db236eb 100644 --- a/docs/vrs/05-harness-state/spec.md +++ b/docs/vrs/05-harness-state/spec.md @@ -7,10 +7,13 @@ spec's R08 section; terminal delivery remains in ## Status -Draft. The envelope below is implemented (`src/harness_state.rs`); producers -and exposure land incrementally under -[`DELTA-005`](../.delta/DELTA-005-harness-state-specified-ahead-of-full-implementation.md). -Open questions are tracked in [open-questions.md](./open-questions.md). +Draft. The envelope (`src/harness_state.rs`), all four producers, the scoped +delivery-input watcher, and the roster/Doctor exposure are implemented; the +former DELTA-005 fence is resolved and deleted, and the DQ-H1 and DQ-H6 +captures are taken and folded in. Draft status remains for the genuinely +unmet residuals: root `DQ3`'s supervisor-following gate (`DQ-H5`) and +Claude's eventless deny path (the remaining `DQ-H1` window). Open questions +are tracked in [open-questions.md](./open-questions.md). ## Scope @@ -57,7 +60,9 @@ One JSON object, atomically written (tmp sibling + rename), newline-terminated: "inputBuffer": "empty | nonempty | unknown", "reason": "", "exit": "", - "ptySession": "", + "ptySession": "", + "incarnation": "", + "seq": 7, "sinceMs": 1787690000000, "writtenAtMs": 1787690300000, "transitions": 41 @@ -70,9 +75,11 @@ Field rules, matching `src/harness_state.rs`: (`DQ-H3`). Readers decode both, plus unrecognized future words as `unknown`. - `blockedOn` unrecognized words decode indeterminate, never `none` — a v2 axis value must not read as "not blocked". -- `ask` names the kind of human ask, machine-readably, while `blockedOn` is - `human` (`none` otherwise; unrecognized words decode indeterminate) — the - axis consumers filter on, so nothing branches on diagnostic `reason`. +- `ask` names what kind of human ask holds the harness, machine-readably — + the axis consumers filter on (`reason` stays diagnostic). Meaningful only + while `blockedOn` is `human`; writers emit `none` otherwise, records from + writers predating the axis default to `none`, and unrecognized future words + decode indeterminate. - `incarnation` is the writing session's token and `seq` its monotonic ownership sequence. A claim is a WRITTEN act under the record lock: the session starting up writes an exitless `ended (superseded)` takeover @@ -83,8 +90,9 @@ Field rules, matching `src/harness_state.rs`: monotonic even when the record itself is unreadable. Racing claimers therefore mint distinct sequences, and a predecessor's still-fresh live record is superseded at relaunch, where the - pty-name-based probe cannot tell sessions apart; the seat reads - `ended (superseded)` until the session's first real observation. Ownership + pty-name-based probe cannot tell sessions apart; readers derive + indeterminate (`claimed`) from the fresh placeholder — a fence is not an + observation — until the session's first real observation. Ownership — coalescing, heartbeat eligibility, terminal suppression (which applies only to exit-bearing terminal records, never the claim placeholder) — is token equality with the claim's direction: a straggler whose claim is @@ -103,10 +111,37 @@ Field rules, matching `src/harness_state.rs`: inherited from beyond the future-skew trust bound) so it stays byte-distinct even against a same-millisecond predecessor. - `reason` is diagnostic only; no consumer branches on it. +- `ptySession` fences every live state: a writer refuses a live observation + that names no session (only `ended` may omit it), because a live record the + probe cannot check would stay definite straight through an external SIGKILL. - `sinceMs` is when the current state was entered and survives heartbeat re-stamps; `writtenAtMs` is the heartbeat. `transitions` is a monotonic counter continued across writer restarts; with `writtenAtMs` it keeps every write byte-distinct. +- Writes are session-owned by incarnation token, and ownership has a + DIRECTION through `seq`. A claim is a WRITTEN act under the record lock — + an exitless `ended (superseded)` takeover record carrying the new token + and a sequence one above the highest the seat has seen (the on-disk + record's or the `.harness-state.seq` floor sidecar's, whichever is + higher, so an unreadable record does not restart the sequence) — so + racing claimers serialize and mint + DISTINCT sequences (no tie exists), and a predecessor's still-fresh live + record is superseded at relaunch; readers derive indeterminate (`claimed`) + from the fresh placeholder until the session's first real observation. A + straggler from a superseded session (its claim below the on-disk sequence) + is refused in live and terminal paths alike, a token-only writer never + claims (it adopts its own session's records, starts virgin ones, and is + refused against foreign tokens), and terminal suppression applies only to + exit-bearing records — never the claim placeholder. `sinceMs` never spans + a restart; a lingering predecessor can neither heartbeat nor overwrite its + successor's record. + Heartbeats and coalescing never touch a record whose schema or token the + writer does not own; a foreign record is left byte-identical by heartbeats + and replaced only by a claiming observation. One residual, stated: + Claude's exported ownership pair is visible to its tool children like + every other hook-environment value (`ST_AGENT` included) — pi stashes and + unexports instead because pi fronts its own environment onto every bash + child, an architectural difference, not an oversight. - Deserialization is additive-tolerant (no `deny_unknown_fields`): a reader may be older than its writer. @@ -121,18 +156,36 @@ What a reader reports, in evaluation order: | Evidence | Reads as | Reason | | --- | --- | --- | | No record file | no observation (`null`) | never observed ≠ `unknown` | +| File exists but cannot be read | `unknown` | `unreadable-record`; an IO error is indeterminate, never absence | | Unparseable / non-v1-shaped bytes | `unknown` | `malformed-record`; never falls back to mtime | +| `schema` is not `st2.harness-state.v1` | `unknown` | `unsupported-schema`; a future schema's words may be spelled like this version's while meaning something else | | `writtenAtMs` > now + 60 s | `unknown` | `future-skew` | | `writtenAtMs` ≤ now − 15 min | `unknown` | `stale` | | Literal `unknown` state (never written by this crate) | `unknown` | `literal-unknown` | | Live state, same-host probe proves `ptySession` dead | `unknown` | `session-dead` | +| Live state, probe available, record names no `ptySession` | `unknown` | `unfenced-record`; nothing to check is not the same as checked | | Live state, probe indeterminate | the recorded state | unprovable evidence downgrades nothing | +| Fresh claim placeholder (`ended`, exitless, reason `superseded`) | `unknown` | `claimed`; a fence is not an observation | | `ended`, any probe result | `ended` | a terminal record outlives its writer | | Otherwise | the recorded tuple | — | Every `unknown` row routes through one constructor and blanks every axis; there is no path from any absence to a definite state. +Two reader-side limits, stated before anyone finds them: `pty kill` removes +the session pidfile, so the liveness probe reads *indeterminate* rather than +*dead* after it — a seat that was `active` when killed genuinely reads +`active` (measured), with Doctor silent, until the staleness horizon or the +next session's written claim supersedes the orphan at relaunch, whichever +comes first. The cross-check is a narrowing of the ungraceful-death window +(provably dead sessions: pidfile present, process gone), not its closure — +OHS-T04/OHS-R07 say exactly this, and no death tombstone is attempted: the +kill that removes the registry entry leaves nothing behind to prove death +with, and fabricating evidence is the one thing this design never does. And +hosts running codex-cli at or above 0.148 produce no Codex observed state at +all: `SUPPORTED_CODEX_CLI_VERSIONS` refuses the launch, correctly, until the +pin moves (#267). + ## Codex producer (OHS-R05) The projection reads the state the control pump already maintains; it adds no @@ -156,7 +209,24 @@ complement of steerable, a delivery predicate (decision 0001's boundary). `inputBuffer` is `unknown` from this producer: the control stream does not see the composer. The projection test must be behavioral — a table that would pass -with every row mapped to `unknown` is not an oracle (#268 §B). +with every row mapped to `unknown` is not an oracle (#268 §B). The wrapper +honors st2's stop signal through every startup phase (socket connect, +initialize, thread binding), not only the bound monitor loop: a stop before +the TUI exists ends the launch gracefully and leaves no record — nothing was +observed — while a stop after it exits through the ordinary terminal-write +path — and the stop handler installs before ANY child is spawned, the +hook-trust preflight's detached app-server included, with the stop flag +polled between short socket timeouts through every startup wait (connect, +the initialize handshake, thread binding, and the preflight's projection +read), so a stop in that window cannot leak a server around a dead wrapper +or sit out a 30-second handshake. Observability never kills a +launch: a claim that cannot be written degrades to a token-only writer with +a warning, a TUI that fails to spawn writes a real terminal record (ended, +launch-error) over the claim placeholder, and a projected transition whose +record write fails does not count as evidence: it is retained as pending and +retried on EVERY pump pass (only the heartbeat is presence-cadence work), so +a stale on-disk state is never kept fresh in contradiction of the latest +observation. ## Claude producer (OHS-R05, OHS-R06) @@ -166,7 +236,8 @@ prompt or tool activity writes `active`; `Stop` writes `idle`; classified from the payload's `tool_name` (`AskUserQuestion` → `question`, anything else → `permission`) — (its meaning is specifically "a human is about to be asked" — it fires only under permission -modes that ask). Events carrying `agent_id` are subagent-nested and never move +modes that ask), classifying its ask kind from the payload's `tool_name`: +`AskUserQuestion` is a `question`, anything else a `permission`. Events carrying `agent_id` are subagent-nested and never move top-level state. The blocked *exit* edge is the next `PreToolUse`, `PostToolUse`, or `Stop` — measured-correct, not merely conservative: the 2026-08-23 batched-permission capture (`DQ-H1`) shows tool execution @@ -185,15 +256,44 @@ driver-declared seats, so both surfaces register identical hooks and a test fails if they drift. A live seat converges without disruption: render output is not part of the launch fingerprint, the merged settings land on the next materialization pass, and the hooks take effect at the next session start. +The registration merges with `arrays="union"`: user-declared hook entries +survive every materialization, while st2's own prior entries — recognizable +by the hook root they reference — are superseded on a hook-set upgrade +rather than accumulated. Hook writes carry the wrapper's exported +incarnation token, so a hook finishing after the wrapper reaped Claude +cannot replace the terminal record. Legacy `deliver "mcp"` seats render the same +canonical registration — they run under claude-session too, and a wrapper +that claims and ends a record nobody transitions would be worse than no +record. A hooks-only seat (the maintained hand-authored example launches +claude directly) gets transitions and blocked-on-you but no heartbeat owner +and no terminal record: its `SessionStart` performs the written claim so +session succession works — eligibility and the takeover are ONE act under +the record lock, so a hooks-only SessionStart racing a wrapper's startup +cannot steal its sequence, and a wrapper's fresh claim placeholder counts +as owned (an abandoned one ages into claimability); a wrapperless claimer +otherwise supersedes only fellow wrapperless tokens, real exit-bearing +terminal records, and staleness — while a live-but-idle seat still ages to +`unknown` and an exit leaves its last state to age out — indeterminate, +never wrong. ## pi producer (OHS-R05, OHS-R08) -The injected extension observes `agent_start`/`agent_end` — a positive, -evented, in-process signal — and reports transitions over the existing -channel; the wrapper writes the record and owns heartbeat and terminal writes. -This is the first producer that satisfies root `DQ2`'s "stronger evented -signal" clause for any harness. pi's composer offers nothing to scrape, so -`inputBuffer` stays `unknown`. +The injected extension observes `agent_start`/`agent_settled` — a positive, +evented, in-process signal; the idle edge is deliberately `agent_settled`, +because `ctx.isIdle()` is still false through `agent_end` and a queued +follow-up turn starts exactly at that boundary (measured against the repo's +own pi captures). Ownership splits by phase rather than living in one +process: the channel owns the live record and its heartbeat, because it is +the one process that sees pi's turn events and its stdio EOF bounds its +evidence, while the wrapper owns only the terminal record — through a terminal-only +observer that never heartbeats but does write `ended` *before* the stop +path's SIGKILL escalation of its own group, the one write that makes an +escalated stop observable at all — because it is the +one process that sees the provider die — and the channel drops queued live +frames once a terminal record is on disk, so the wrapper's write is the +incarnation's last word. This is the first producer that satisfies root +`DQ2`'s "stronger evented signal" clause for any harness. pi's composer +offers nothing to scrape, so `inputBuffer` stays `unknown`. ## OpenCode producer (OHS-R08) @@ -204,23 +304,46 @@ allocates a loopback port and a per-seat password, launches the TUI bound to them, owns the presence lease and the observed-state record, and projects the `/event` SSE stream: -| Signal | state | blockedOn | reason | -|---|---|---|---| -| `session.status {type: busy}` | `active` | `none` | — | -| `session.status {type: retry}` | `active` | `none` | `retry` | -| `session.status {type: idle}` / `session.idle` | `idle` | `none` | — | -| `permission.asked` … `permission.replied` (same ask id; spelled `id` on entry, `requestID` on exit — measured) | `active` | `human` | `permission` | -| `question.asked` … `question.replied\|rejected` (same ask id, same spelling split) | `active` | `human` | `question` | -| `session.error {ProviderAuthError}` | `ended` | `none` | `providerAuth` | -| `session.error` (other arms) | `idle` | `none` | `error:` | -| child exit / stop path | `ended` | `none` | exit status | +| Signal | state | blockedOn | ask | reason | +|---|---|---|---|---| +| `session.status {type: busy}` | `active` | `none` | `none` | — | +| `session.status {type: retry}` | `active` | `none` | `none` | `retry` | +| `session.status {type: idle}` / `session.idle` | `idle` | `none` | `none` | — | +| `permission.asked` … `permission.replied` (same ask id; spelled `id` on entry, `requestID` on exit — measured) | `active` | `human` | `permission` | `permission` | +| `question.asked` … `question.replied\|rejected` (same ask id, same spelling split) | `active` | `human` | `question` | `question` | +| `session.error {ProviderAuthError}` | `ended` | `none` | `none` | `providerAuth` | +| `session.error` (other arms) | `idle` | `none` | `none` | `error:` | +| child exit / stop path | `ended` | `none` | `none` | exit status | A dedicated seat aggregates across the server's sessions: any busy session is activity, any open ask is a human block, and idle requires positive level evidence (`/session/status` omits idle sessions, so an empty map over a live -server is the idle proof, re-read on every SSE (re)connect). A dropped stream -stops the heartbeat; `inputBuffer` stays `unknown` — the `/tui/*` surface is -write-only. +server is the idle proof, re-read on every SSE (re)connect). Asks open across +a reconnect are recovered from both pending listings — `GET /permission` and +`GET /question`, each measured on 1.18.19 — with their ids kept so the +id-matched exits still release them; the seed builds a fresh projection and +swaps it in whole only when the status level (which must be the documented +object shape — null or an array proves nothing), every status word, and both +listings all read cleanly — a listing entry without a readable id is a +pending ask the id-matched exit could never release, so it fails the seed +rather than being skipped. A mid-seed failure leaves nothing half-seeded, a +successful re-seed clears stale entries whose exits passed during the +outage, and otherwise evidence stays off and the seed retries. An +unrecognized `session.status` word on ANY session poisons the projection +outright — a tracked-busy entry can no longer be trusted to clear, and an +untracked session in a state this version cannot read makes standing idle +evidence a fabrication — so non-terminal observations are withheld (a +sticky terminal still outranks the poison) and evidence drops until a fresh +seed replaces the picture. An SSE drop marks the observation +stream interrupted, so the first post-reseed observation opens a fresh +transition rather than claiming continuity across the outage. The `/doc` +subset gate names every consumed arm, exit events and pending listings +included, so a release renaming an exit is refused up front rather than +holding `blockedOn: human` forever. A dropped stream stops the heartbeat, the +stream carries a silence horizon of at least twice the measured heartbeat +cadence so a stalled socket surfaces as a disconnect instead of keeping +evidence alive forever, and `inputBuffer` stays `unknown` — the `/tui/*` +surface is write-only. The native delivery transport mirrors the Codex FIFO discipline: an `Attempted` receipt persisted before transport, `POST @@ -232,8 +355,14 @@ reads back durably from the server. The `/tui/append-prompt` and two gates: a `SUPPORTED_OPENCODE_VERSIONS` pin and a live `/doc` OpenAPI subset check naming every arm st2 consumes; observation runs behind the `/doc` check alone, since its vocabulary already degrades to indeterminate. -Deliveries target the most recently observed session; a seat whose TUI has not -yet created one waits rather than creating sessions itself. +Deliveries target the most recently observed session; a seat whose TUI has +not yet created one waits rather than creating sessions itself, and a session +that settled before the observer connected — invisible to events and to +`/session/status` alike — is recovered from the session listing when work is +pending. The status seed trusts exactly the pinned words (`busy`, `retry`, +`idle`) and fails closed on anything else, and the stop path rewrites its +pre-signal escalation cover with the exit the grace-window reap actually +observed. ## Exposure (OHS-R09, OHS-R10) @@ -254,7 +383,11 @@ yet created one waits rather than creating sessions itself. `null` when no record exists. The derivation above is already applied — a consumer never re-implements staleness. Roster reads pass the same-host -liveness probe only for agents whose resolved host is this host. `status`, +liveness probe only for agents whose resolved host is this host, resolving +the pty root exactly as the runner does (`PTY_ROOT`, else the catalog's own +pty root; the legacy `PTY_SESSION_DIR` is deliberately not honored — the +runner never uses it, and probing a directory st2-managed sessions never +touch turns provable deaths into indeterminate reads). `status`, `desiredState`, and `lastActivity` keep their exact meanings; the three full-string pinned assertions and the stable-roster invariant wording are updated deliberately in the change that adds the field. Doctor prints an diff --git a/docs/vrs/ontology.md b/docs/vrs/ontology.md index 33d90b66..436c1b03 100644 --- a/docs/vrs/ontology.md +++ b/docs/vrs/ontology.md @@ -121,7 +121,8 @@ Authority: [`reconcile::Session`](../../src/reconcile.rs#L16-L26) The driver-written record of what a harness is seen doing: activity (`idle`/`active`/`child`/`ended`, with `unknown` derived and never written), -who it is blocked on, and what its input buffer holds. The observed +who it is blocked on — and, when blocked on a human, what kind of ask holds +it (`permission`/`question`/`review`) — and what its input buffer holds. The observed counterpart of the declared axes: it is not [presence](#presence) (agent- authored availability), not [session state](#session-state) (task-record liveness), not R08's *declared activity status*, and not R09's *working @@ -350,7 +351,7 @@ explicit `unpark` recovery request. [session state](../../src/reconcile.rs#L16-L26) for runtime liveness. Avoid bare *agent status* when either could be meant. - Use [observed harness state](05-harness-state/requirements.md) for the - driver-observed working signal. It is a third axis beside presence and + driver-observed activity signal. It is a third axis beside presence and session state: R08's *activity status* stays the declared, agent-authored signal, and neither axis rewrites the other. Bare *activity* does not identify which is meant. diff --git a/tests/harness_state_teardown.rs b/tests/harness_state_teardown.rs new file mode 100644 index 00000000..b78b942e --- /dev/null +++ b/tests/harness_state_teardown.rs @@ -0,0 +1,242 @@ +//! The stop path's terminal-record ordering, proven against the real wrapper binary. +//! +//! st2's own stop escalates to a SIGKILL of the wrapper's process group after `STOP_GRACE`, which +//! the wrapper itself cannot survive — so the observed-harness-state terminal record must land +//! *before* that escalation. No in-process test can cover this (the escalation kills the test's +//! own group), so the wrapper runs here as a real child in its own process group. + +#![cfg(unix)] + +use std::fs; +use std::os::unix::fs::PermissionsExt; +use std::os::unix::process::CommandExt as _; +use std::path::Path; +use std::process::{Command, Stdio}; +use std::time::{Duration, Instant}; + +fn executable(path: &Path, body: &str) { + fs::write(path, body).unwrap(); + fs::set_permissions(path, fs::Permissions::from_mode(0o755)).unwrap(); +} + +#[test] +fn stop_escalation_writes_the_terminal_record_before_sigkill() { + let tmp = tempfile::tempdir().unwrap(); + let catalog = tmp.path().join("catalog"); + let declaration = catalog.join("agents/h/worker/agent.kdl"); + fs::create_dir_all(declaration.parent().unwrap()).unwrap(); + fs::write( + &declaration, + r#"agent "worker" { host "h"; command "true" }"#, + ) + .unwrap(); + let agent_dir = declaration.parent().unwrap().to_path_buf(); + let bin = tmp.path().join("bin"); + fs::create_dir_all(&bin).unwrap(); + executable(&bin.join("hostname"), "#!/bin/sh\necho h\n"); + let path = format!( + "{}:{}", + bin.display(), + std::env::var("PATH").unwrap_or_default() + ); + + // The provider ignores SIGTERM, so the wrapper's grace window expires and it must escalate. + let mut wrapper = Command::new(env!("CARGO_BIN_EXE_st2")); + wrapper + .args(["--catalog"]) + .arg(&catalog) + .args([ + "driver", + "claude-session", + "--identity", + "worker", + "--runtime-id", + "worker", + "--", + "sh", + "-c", + // The ready marker synchronizes the signal: SIGTERM may only fly once the trap is + // provably installed, or a slow scheduler lets the child die in the grace window and + // the escalation path goes untested. + "trap '' TERM; : > \"$READY_MARKER\"; sleep 60", + ]) + .env("READY_MARKER", tmp.path().join("provider-ready")) + .env("PATH", &path) + .stdin(Stdio::null()) + .stdout(Stdio::null()) + .stderr(Stdio::null()); + unsafe { + // Its own process group: the wrapper SIGKILLs -pgid on escalation, and that must never + // reach this test process. + wrapper.pre_exec(|| { + libc::setsid(); + Ok(()) + }); + } + let mut wrapper = wrapper.spawn().unwrap(); + + // The provider proves its TERM trap is installed before the wrapper is signaled. + let ready = tmp.path().join("provider-ready"); + let started = Instant::now(); + while !ready.exists() { + assert!( + started.elapsed() < Duration::from_secs(10), + "provider never installed its trap" + ); + std::thread::sleep(Duration::from_millis(25)); + } + + unsafe { + libc::kill(wrapper.id() as i32, libc::SIGTERM); + } + // STOP_GRACE is 5 s; the wrapper dies with its own SIGKILL shortly after — and the exit + // status must SHOW that: a wrapper that exited any other way never exercised escalation. + let deadline = Instant::now() + Duration::from_secs(15); + let wrapper_exit = loop { + if let Some(exit) = wrapper.try_wait().unwrap() { + break exit; + } + assert!( + Instant::now() < deadline, + "wrapper survived its own escalation" + ); + std::thread::sleep(Duration::from_millis(50)); + }; + assert_eq!( + { + use std::os::unix::process::ExitStatusExt as _; + wrapper_exit.signal() + }, + Some(libc::SIGKILL), + "the wrapper must die to its own SIGKILL escalation, exit: {wrapper_exit:?}" + ); + + let raw = fs::read_to_string(agent_dir.join("harness-state")) + .expect("terminal record must land before the SIGKILL escalation"); + let record: serde_json::Value = serde_json::from_str(&raw).unwrap(); + assert_eq!(record["state"], "ended", "record: {raw}"); + assert_eq!(record["exit"], "signal 9", "record: {raw}"); +} + +/// The OpenCode wrapper has its own stop implementation; its escalation cover must land before +/// the group SIGKILL exactly like the shared wrapper's. +#[test] +fn opencode_stop_escalation_writes_the_cover_record_before_sigkill() { + let (record, mut wrapper, _tmp) = + spawn_opencode_wrapper("trap '' TERM; : > \"$READY_MARKER\"; sleep 60"); + unsafe { + libc::kill(wrapper.id() as i32, libc::SIGTERM); + } + let deadline = Instant::now() + Duration::from_secs(15); + let wrapper_exit = loop { + if let Some(exit) = wrapper.try_wait().unwrap() { + break exit; + } + assert!( + Instant::now() < deadline, + "wrapper survived its own escalation" + ); + std::thread::sleep(Duration::from_millis(50)); + }; + assert_eq!( + { + use std::os::unix::process::ExitStatusExt as _; + wrapper_exit.signal() + }, + Some(libc::SIGKILL), + "the wrapper must die to its own SIGKILL escalation, exit: {wrapper_exit:?}" + ); + let raw = fs::read_to_string(&record).expect("cover record must land before the SIGKILL"); + let value: serde_json::Value = serde_json::from_str(&raw).unwrap(); + assert_eq!(value["state"], "ended", "record: {raw}"); + assert_eq!(value["exit"], "stopped", "record: {raw}"); +} + +/// A provider that yields inside the grace window leaves the wrapper alive, and the record is +/// rewritten with the exit the reap actually observed — never left as the "stopped" cover. +#[test] +fn opencode_graceful_stop_records_the_real_reaped_exit() { + let (record, mut wrapper, _tmp) = spawn_opencode_wrapper(": > \"$READY_MARKER\"; sleep 60"); + unsafe { + libc::kill(wrapper.id() as i32, libc::SIGTERM); + } + let deadline = Instant::now() + Duration::from_secs(15); + loop { + if wrapper.try_wait().unwrap().is_some() { + break; + } + assert!( + Instant::now() < deadline, + "wrapper never finished its graceful stop" + ); + std::thread::sleep(Duration::from_millis(50)); + } + let raw = fs::read_to_string(&record).unwrap(); + let value: serde_json::Value = serde_json::from_str(&raw).unwrap(); + assert_eq!(value["state"], "ended", "record: {raw}"); + assert_eq!(value["exit"], "signal 15", "record: {raw}"); +} + +/// Spawn the real opencode-session wrapper in its own process group over a shell provider (the +/// API gate simply never opens — presence and the terminal path are what these prove), waiting on +/// the provider's ready marker before returning. +fn spawn_opencode_wrapper( + provider_script: &str, +) -> (std::path::PathBuf, std::process::Child, tempfile::TempDir) { + let tmp = tempfile::tempdir().unwrap(); + let catalog = tmp.path().join("catalog"); + let declaration = catalog.join("agents/h/worker/agent.kdl"); + fs::create_dir_all(declaration.parent().unwrap()).unwrap(); + fs::write( + &declaration, + r#"agent "worker" { host "h"; command "true" }"#, + ) + .unwrap(); + let agent_dir = declaration.parent().unwrap().to_path_buf(); + let bin = tmp.path().join("bin"); + fs::create_dir_all(&bin).unwrap(); + executable(&bin.join("hostname"), "#!/bin/sh\necho h\n"); + let path = format!( + "{}:{}", + bin.display(), + std::env::var("PATH").unwrap_or_default() + ); + let mut wrapper = Command::new(env!("CARGO_BIN_EXE_st2")); + wrapper + .args(["--catalog"]) + .arg(&catalog) + .args([ + "driver", + "opencode-session", + "--identity", + "worker", + "--runtime-id", + "worker", + "--", + "sh", + "-c", + provider_script, + ]) + .env("PATH", &path) + .env("READY_MARKER", tmp.path().join("provider-ready")) + .stdin(Stdio::null()) + .stdout(Stdio::null()) + .stderr(Stdio::null()); + unsafe { + wrapper.pre_exec(|| { + libc::setsid(); + Ok(()) + }); + } + let wrapper = wrapper.spawn().unwrap(); + let ready = tmp.path().join("provider-ready"); + let started = Instant::now(); + while !ready.exists() { + assert!( + started.elapsed() < Duration::from_secs(10), + "provider never signalled ready" + ); + std::thread::sleep(Duration::from_millis(25)); + } + (agent_dir.join("harness-state"), wrapper, tmp) +}