Skip to content

docs(vrs): record observed harness state experiments - #263

Closed
schickling wants to merge 2 commits into
mainfrom
schickling/2026-08-17-pi
Closed

docs(vrs): record observed harness state experiments#263
schickling wants to merge 2 commits into
mainfrom
schickling/2026-08-17-pi

Conversation

@schickling

@schickling schickling commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Why

st2 cannot see what a harness is doing. Presence is agent-declared — status::refresh proves only that a process is alive and preserves whatever the model last wrote — so a wedged agent and a working one are indistinguishable for up to fifteen minutes. Meanwhile CodexObservedState (src/codex_app_server.rs:166) already models exactly this concept for one harness, typed and persisted under st2.codex-control-state.v1, in a hash-keyed host-local file that nothing outside that module reads. This is the activity-lease slice of #162 — "activity leases (idle, active, child, unknown)" — measured before being specified, as DQ3 requires.

These are records of what was measured, not a specification. No requirements text, no ontology entry, no code.

What

  • Record that Codex already carries blocked-on-human on the wire: ThreadStatus.active has a required activeFlags field, and observe_thread_status (:770) reads only the status word.
  • Record that Claude can prove the same signal: the binary carries 31 hook events against 10 documented, and PermissionRequest fires only when a human is about to be asked — in 2 of 9 captures, and in zero bypassPermissions/auto runs.
  • Record that Claude death is silent for SIGTERM and SIGKILL alike, and that a wrapper-owned terminal write does not close the gap, because stop_provider_group escalates into the wrapper's own process group.
  • Record that a shared append-only history conflicts under Git while host-named segments do not, and that retention disagreement between hosts resolves to the minimum.
  • Record the storage question as unresolved: the transport was not identified, so no cost model is claimed.
  • Propose 0004, asserting only the half that survived independent adversarial review: observed state is inadmissible as transport authorization.

Impact

Nothing changes. This adds five documents and their captures under docs/vrs/, and nothing else.

  • No source file is modified. No requirements, ontology, or spec text is touched.
  • INVARIANTS.md is unchanged; no invariant is claimed, relaxed, or added.
  • No Agent Spec field is proposed. Agent Spec authoring authority remains with compoundingtech/evals.
  • 0004 is Status: proposed and explicitly says it is not accepted.

Checks

git diff --check origin/main..HEAD
  → trailing whitespace reported ONLY in docs/vrs/.experiments/2026-08-17-captures/**/*.raw
    These are verbatim PTY screen captures; the trailing bytes are part of the evidence.

nix develop --command cargo test --locked --test invariants
  → test qualified_proof_references_resolve ... ok
    test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

No Darwin run: no code changed.

Limits, stated up front

  • activeFlags is unverified on a supported version. Corrected 2026-08-18: schemas generated from 0.145.0 and 0.146.0 binaries — both already admitted by SUPPORTED_CODEX_CLI_VERSIONS — carry ActiveThreadStatus with required: ["activeFlags","type"] and an identical two-value ThreadActiveFlag. The field is available on a supported version today and no finding here depends on admitting 0.147.0. What stands: a populated activeFlags has never been observed on the wire on any version — the live capture confirmed only "activeFlags":[] before the account hit its usage limit — so the steer-into-approval consequence remains inferred, not observed.
  • The prototype PTY observer is broken for Claude and is not proposed here. expected = "" collides with the Claude arm's representation of an empty composer, so it proves idle once and reports unproven thereafter.
  • DQ3's supervisor-following half is unproven. Only stale-state behavior was measured.
  • The CodexHoldReason mapping is lossy, not residue-free: four of six rows report unknown, two of which Codex positively reported as active.

Ready for review.

Related

Four experiment records and one draft decision record for observed harness
state: a coarse per-agent lifecycle signal, produced by a harness driver and
consumed generically, kept separate from agent-declared presence.

These are records of what was measured, not a specification. No requirements
text, no ontology entry, no code.

Measured and captured:

- Codex already carries blocked-on-human on the wire. `ThreadStatus.active`
  has a required `activeFlags` field; `observe_thread_status` reads only the
  status word. Read from the 0.147.0 schema, which st2 does not yet support.
- Claude can prove the same. The binary carries 31 hook events against 10
  documented; `PermissionRequest` fires only when a human is about to be
  asked. Nine captures are included.
- Claude death is silent for SIGTERM and SIGKILL alike, and a wrapper-owned
  write does not close the gap: `stop_provider_group` escalates into the
  wrapper's own process group.
- A shared append-only history conflicts under Git; host-named segments do
  not. Retention disagreement between hosts resolves to the minimum.
- The transport was not identified. Fabric is unobservable on this host, so
  no cost model is claimed.

The decision record is `Status: proposed` and asserts only the half that
survived independent review: observed state is inadmissible as transport
authorization. The scheduling half is withdrawn to an open question.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@schickling schickling added area:driver Harness drivers: launch, MCP, app-server, native delivery · Set: manual area:vrs VRS documentation system (vision/requirements/spec) · Set: manual harness:claude Claude Code-specific behavior · Set: manual harness:codex Codex-specific behavior · Set: manual type:docs Documentation-only change or documentation task · Set: manual labels Aug 18, 2026
The experiment recorded that `activeFlags` was verified only on codex-cli
0.147.0, and inferred that consuming it required admitting that version.

Schemas generated from 0.145.0 and 0.146.0 binaries — both already admitted —
carry `ActiveThreadStatus` with `required: ["activeFlags","type"]` and the same
two-value `ThreadActiveFlag`. The field is available on a supported version
today, so no finding depends on the version pin.

What is unchanged: no populated `activeFlags` has been observed on the wire on
any version, so the steer-into-approval consequence remains inferred.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@schickling
schickling marked this pull request as ready for review August 18, 2026 07:00
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@schickling-assistant

Copy link
Copy Markdown
Contributor

Superseded by the merged observed-harness-state implementation (#319#326). The experimental conclusions are now represented by the accepted architecture.

Posted on behalf of @schickling
field value
agent_identity dev3.direct.omp.jmg9x29x
session dev3.jmg9x29x
agent_persona generalist
agent_supervisor unavailable
agent_tool OMP
agent_tool_version 18.0.11
agent_runtime OMP 18.0.11
tooling_profile dotfiles@2161b9c

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:driver Harness drivers: launch, MCP, app-server, native delivery · Set: manual area:vrs VRS documentation system (vision/requirements/spec) · Set: manual harness:claude Claude Code-specific behavior · Set: manual harness:codex Codex-specific behavior · Set: manual type:docs Documentation-only change or documentation task · Set: manual

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants