Design question: intended input boundary for embedded PTY clients

[schickling-assistant]

@myobie, I would appreciate a design check before we extend the embedding protocol further: where is key interpretation intended to live for a terminal UI embedding a PTY session?

The concrete consumer is Fractal, which embeds a live session inside another terminal application. We have two independent findings:

• Alt+Left arrives from the outer terminal as ESC b, and direct pty attach delivers those bytes unchanged. Fractal currently reads a semantic Crossterm event and reconstructs only b, so that loss is in the embedding client rather than PTY.
• PTY's interactive attach path is not byte-transparent today. It converts input Buffer -> string -> Buffer and scans 0x1c for detach. At current main (025a9034), bytes 1b 62 ff 00 become 1b 62 ef bf bd 00.

Our current candidate design is:

1. The embedding client owns decoding of its outer terminal input and keeps both exact raw bytes and a normalized semantic event.
2. A mode-aware codec reuses raw bytes only when the outer and child input modes are compatible; otherwise it re-encodes the semantic event for the child.
3. PTY remains a byte-oriented session transport. Detach is an explicit control message, never inferred by scanning terminal data.
4. Attach admission, initial geometry/mode/screen state, updates, and the terminal outcome are ordered on one generation-bound connection.

Before implementing that, is this the intended responsibility split, or is there a simpler supported pattern we should use?

In particular:

• Should a non-TypeScript embedder speak the existing daemon packet protocol directly (the conceptual equivalent of SessionConnection) and add only the missing byte-safe/generation-bound pieces?
• Or should machine attach grow one bidirectional framed channel so embedders never need to own the daemon protocol?
• Should PTY expose only ordered child-mode state while the embedder owns key decoding/re-encoding, or is PTY intended to own more of that translation?
• Is an explicit detach frame the intended replacement for the interactive CLI's Ctrl-\\ byte interception in an embedded path?
• Should older daemons reject the new embedded contract explicitly, without automatic restart or a legacy fallback?

This is a design question, not yet a request for a particular v2 protocol. Related work already covers incarnation binding (#121), mode visibility (#130), ordered effective geometry (#134), terminal detach outcomes (#151), and PTY's named-key encoder (#13). I would like to avoid introducing a parallel abstraction if those are meant to compose into a simpler embedding API.

Current direction (2026-08-03)

The investigation changed the priority, but not the responsibility split above. pty-layout is strong evidence that the current PTY protocol already supports a useful compositor locally and through pty-relay over SSH and WebSocket. The first step is therefore to improve and test clients against the current protocol. Machine attach v2 is not a prerequisite for making Fractal useful.

• Keep PTY as the byte-framed session transport.
• Let compositors own outer-terminal negotiation, local focus, shortcuts, and mode-aware key encoding.
• Make regular pty attach and embedded clients share the same tested input behavior where possible, as Nathan suggested.
• Preserve existing Kitty, Ghostty, relay, resize, and terminal-cleanup behavior through black-box fixtures.
• Use Specify terminal DATA byte semantics and add Node/Rust conformance fixtures #160 to specify UTF-8 text versus byte semantics and run shared Node/Rust/client fixtures.
• Use Define Node compatibility contract and export an embeddable PTY client crate pty-rust#1 and ERR_UNSUPPORTED_NODE_MODULES_TYPE_STRIPPING when running pty CLI #3 for the Rust compatibility target and one attached/spawned terminal handle.
• Add protocol behavior only after a failing fixture proves the current messages cannot express the requirement.

The original v2 questions remain useful as possible answers to proven ordering or lifecycle gaps. They are no longer assumptions that block current-protocol client work.

[schickling-assistant]

@myobie, the first exact Fractal -> PTY v2 composition test exposed one more architectural question that blocks exact admission:

• OPEN requires expectedGeneration so admission cannot race onto a replacement daemon.
• pty list --json exposes createdAt but not the opaque daemon generation.
• Fractal therefore had no public discovery source for the required token and (incorrectly) supplied createdAt; the fresh daemon rejected admission atomically, as designed.

The clean options seem to be:

1. Make generation an explicit opaque field of the existing public session inventory / list --json contract.
2. Add a machine-specific inventory/descriptor API that returns { sessionId, generation, ... }, keeping human CLI inventory separate.
3. Change admission to mint/resolve a token on the same connection, though that appears to weaken the caller's exact-incarnation intent unless it carries another fence.

My current preference is (1) unless PTY intends a broader typed machine inventory soon: it is the smallest single source of truth, generation is already non-secret lifecycle identity, and the server remains the admission authority. Is that the intended shape, or would you choose (2)?

[schickling-assistant]

Update: draft PR #156 now implements and exercises the proposed boundary end to end.

• embedded and human CLI attach share the generation-bound machine-v2 behavioral core;
• PTY transports framed bytes plus ordered child-mode state and explicit detach/outcomes;
• the embedding client owns outer-terminal decoding and mode-aware re-encoding;
• attach lifecycle and cancellation are one tagged state machine, with causal tests for reconnect, delayed READY/resize, detach races, and byte fragmentation;
• the remaining legacy writable protocol is compatibility-only for existing public/testing/TUI consumers, with no new consumers and an explicit migration/removal gate.

The Nix and Vitest gates are green at 85ce111. I would still value your view on whether this matches PTY's intended architecture or whether there is a materially simpler supported key/input path we should converge on before making the PR ready.

[schickling-assistant]

Final implementation update: PR #156 now also covers xterm modifyOtherKeys=2 input. The shared interactive policy recognizes CSI 27;5;92~ as local Ctrl-\ only while the causally installed mode snapshot says level 2; levels 0/1 and mode transitions preserve bytes exactly, and every chunk partition is covered. Nix and Vitest are green at ea7bc1063bfb16309de5e2717c548c70605839e4.

@myobie, we would still value your judgment on the architectural question in the issue body: is this the intended long-term boundary for embedded and human clients, or is there a simpler PTY-native pattern we should converge on before making v2 stable?

[schickling-assistant]

Resolution:

• Keep the current PTY protocol.
• Improve and test clients first.
• PTY stays a byte-framed session transport.
• Fractal owns focus, shortcuts, and input routing.
• pty-rust owns the reusable streaming input decoder and child-mode encoder for embedded Rust clients.
• Do not add machine attach v2 now.
• Add protocol behavior only after a failing shared fixture proves that it is needed.

Future input-boundary and conformance work is in compoundingtech/pty-rust#3 and compoundingtech/pty-rust#4. The original design questions and tests remain useful evidence.

Closing this design question as answered.