Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions CATALOG.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
<!-- Generated by bin/generate-catalog.sh from cell KDL and evidence/*.tsv. Do not edit. -->
# Current eval catalog and evidence

This is the sole current corpus surface: **58 included cells** (18 model-free,
This is the sole current corpus surface: **59 included cells** (19 model-free,
40 model-backed) and **4 retired exclusions**. Inventory fields are derived from the active
KDL and executable judge scripts. Latest accepted PASS evidence remains distinct from the append-only run
history: a failed last run is visible without being advertised as accepted evidence, and cells with no
Expand Down Expand Up @@ -50,6 +50,7 @@ harness-native loader plus canonical hook file.
| `presence-ding-matrix` | model-free | — | 0 | none | `60s` | 6 | — | **NO STRUCTURED RUN** |
| `pty-attach-machine-stream` | model-free | — | 0 | none | `90s` | 5 | — | **NO STRUCTURED RUN** |
| `pty-attach-only` | model-free | — | 0 | none | `90s` | 7 | — | **NO STRUCTURED RUN** |
| `pty-attach-outcomes-and-roles` | model-free | — | 0 | none | `60s` | 5 | — | **NO STRUCTURED RUN** |
| `pty-send-peek` | model-free | — | 0 | none | `120s` | 5 | **PASS** 2026-07-28T19:53:59Z, 13/13, source [`b7da882`](https://github.com/compoundingtech/evals/commit/b7da8826b503e0408e309af8702fb684d28e3a19), [receipt](evidence/stable-main-phase4-runs-20260728.json) | **PASS** 2026-07-28T19:53:59Z, 13/13, source [`b7da882`](https://github.com/compoundingtech/evals/commit/b7da8826b503e0408e309af8702fb684d28e3a19), [receipt](evidence/stable-main-phase4-runs-20260728.json) |
| `reconcile-retire-keep` | model-free | — | 0 | none | `90s` | 6 | — | **NO STRUCTURED RUN** |
| `render-target-safety` | model-free | — | 0 | none | `90s` | 5 | — | **NO STRUCTURED RUN** |
Expand Down Expand Up @@ -102,6 +103,7 @@ while the last-run column makes a recorded failure distinct from a cell with no
- `hook-integrity` / `cell` — deterministic native hook-installation probe; no live harness seat
- `host-lock-health-negatives` / `cell` — deterministic native doctor negative probe; no harness seat
- `pty-attach-machine-stream` / `cell` — deterministic installed PTY attach composition probe; no harness seat
- `pty-attach-outcomes-and-roles` / `cell` — deterministic installed PTY outcome and raw-socket role-transition probe; no harness seat
- `pty-attach-only` / `cell` — deterministic PTY dead-attach policy probe; no harness seat
- `pty-send-peek` / `cell` — deterministic PTY transport probe; no harness seat
- `presence-ding-matrix` / `cell` — deterministic native presence and DING probe; no harness seat
Expand All @@ -114,7 +116,7 @@ while the last-run column makes a recorded failure distinct from a cell with no
- `targeted-reconcile-isolation` / `cell` — deterministic selected reconcile isolation probe; no bus-connected agent
- `two-networks-coexist` / `cell` — deterministic catalog-isolation probe; no harness seat

These **19** rows are generated from structured exclusions. A cell-level row is required for
These **20** rows are generated from structured exclusions. A cell-level row is required for
every derived model-free cell. The one-shot docs cold reader is a model-backed grader, but not a bus-connected
agent: it has no st2 identity, DING sidecar, or lifecycle hooks.

Expand Down
55 changes: 55 additions & 0 deletions cells/pty-attach-outcomes-and-roles/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# pty-attach-outcomes-and-roles

**Type:** PTY installed-launcher and raw-socket composition. **Upstream
contracts:** the intentional machine-detach outcome and explicit same-socket
client-role replacement proposed against PTY main.

**Capabilities required:** `pty,node`. No model and no bus. The cell executes
the `pty` binary selected by `PATH`; it does not import PTY source modules or
invoke `dist/cli.js`.

## What it proves

- **Intentional outcome:** the shipped `bin/pty` launcher preserves fd 3. Once
its initial baseline is observed, one real Ctrl-\\ input produces one
terminal, empty `DETACH`, clean descriptor EOF, empty stdout/stderr, and
process status zero.
- **Truncation control:** a public administrative `pty kill` of a separate
eval-owned session after its initial baseline closes fd 3 without `EXIT` or
`DETACH`; the attach process must exit nonzero. This distinguishes abrupt
session loss from an intentional local user action.
- **Role promotion:** one raw socket sends `PEEK`, then `ATTACH`, terminal data,
`RESIZE`, and `STATUS`. A separate attached anchor provides an output barrier.
The promoted client becomes writable and changes the shared grid from
`30x100` to `18x60`.
- **Role demotion:** another raw socket sends `ATTACH`, then `PEEK`, followed by
forbidden `DATA` and `RESIZE` plus a `STATUS` barrier. Its constraint is
removed, the grid returns to the anchor's `30x100`, and no forbidden bytes
reach the PTY before a later anchor marker.
- **Oracle mutations:** synthetic observations reject a missing, non-empty, or
wrong detach outcome; false truncation success; failed promotion; wrong
min-grid; leaked demoted input; a retained demoted size constraint; and an
empty session catalog with a live daemon or child PID.
- **Cleanup:** every socket is closed, cleanup commands succeed, the session
catalog is empty, and every captured eval-owned daemon and child PID is dead.

The output barrier is causal rather than sleep-based: the anchor writes only
after the transitioning socket receives `STATUS`, and the PTY must emit the
anchor marker. Any earlier accepted input necessarily precedes that marker.

## Validation revisions

The acceptance cell was proven failing against PTY main `d5fabc3` and passing
against the packaged PTY built from merged upstream main
`9eb958c5aae026d5c05690ab72b528662c55708d`. That revision contains the attach
stream fixture prerequisite from #145, intentional detach outcomes from #146,
explicit same-socket role replacement from #147, and live-daemon registry
recovery from #128. The maintained cell now
runs directly against packaged PTY main; it has no validation-only runtime
dependency.

## Run it

```sh
st2 eval ./cells/pty-attach-outcomes-and-roles/
```
Loading