Declared event streams: agent-owned subscriptions delivering deduplicated world events over the native bus

[schickling]

Summary

Add declared event streams: named, agent-owned subscriptions that turn external-world changes (CI runs, PRs, timers, monitoring) into durable, deduplicated events in the owning agent's inbox, waking it through the existing delivery transports (DING / MCP push / app-server) untouched. This is the design issue that #137's ingress contract anticipated; the full intent layer lands as the docs/vrs/04-stream/ subsystem with two decision records and four executable experiment records.

The design in one screen

agent "demo" {
  driver claude {}
  stream "gh-ci" {
    command "gh-ci-watch --repo compoundingtech/st2"   // supervised adapter
  }
  stream "webhook" {}                                   // no command = external ingress endpoint
}

st2 event emit dev3.demo --stream gh-ci \
  --event-id run-812 --key pr-42 --supersede \
  --subject 'CI: failure on PR #42'

• A stream with a command lowers to one derived exec companion (stream-<name>) through the same seam as the generated DING — inheriting launch-with-agent, suspend/retire teardown, independent parking, crash surfacing to the supervisor. A stream without a command is the Add idempotent shell-driven event ingress over the native message bus #137 external-producer boundary (systemd timers, webhook bridges).
• An event is an ordinary inbox record (same filename grammar, directories, archive semantics) carrying stream / event-id / optional key frontmatter. Old readers see a normal message; delivery paths are untouched; DING gains exactly one branch: the » marker.
• event-id is mandatory and dedup scope is (stream, event-id): replays — concurrent or crash-interrupted — return the original filename and never re-notify. Dedup state is a bounded constant-size ring per stream; events do not write the Agent Sent ledger.
• --supersede archives the stream's unread predecessor per key before publishing (log-compaction semantics), producer-side only — DING staged ownership is never touched.
• Agents self-subscribe by editing their own declaration through the existing serialized authoring path (st2 stream add/rm, R25 authority).
• Adapters (the world-specific logic) ship outside st2 (nix layer); st2 keeps the contract plus no-network example fakes. No cadence in v1: a timer is a sleeping adapter; the reserved schedule node and DQ1 scheduled work stay a separate track.

Why this shape — evidence

Four executable explorations back the two decision records (docs/vrs/.decisions/0004, 0005; experiments under docs/vrs/04-stream/.experiments/):

1. Ingress comparison (16 tests): both the message path and the service-principal path are exactly-once under 12-way concurrent replay; the message path cannot express a non-agent producer and writes one Agent Sent row per event (MESSAGE-R11); coalesce-to-latest cannot live in DING.
2. Lifecycle spike (27 tests): the derived-companion seam carries stream sources with zero changes to run/flapping/park/inventory; a content hash cannot distinguish replay-after-restart from a genuine repeat — the producer-supplied event-id is the contract.
3. Unification-pole design + spike (13 tests): one actor model/one ledger is crash-safe at all nine publication checkpoints even for non-agent senders — but imposes prove-what-I-said ledger machinery (permanent, O(history) validation) on machine producers.
4. Differentiation-pole design + spike (12 tests + DING proofs): producer-side supersession never re-pastes a DING-staged predecessor (24 supersedes in 146 ms → one poke, zero staged retries, one unread head); stream state stays constant-size; R15 holds.

Both poles converged independently on: events ride ordinary inbox files, one new glyph, fan-out fixed by scoping dedup per recipient, and the typed request/reply envelopes eventually absorbed (a reply to an event is an ordinary message reply; pending|replied derives from in-reply-to) — staged behind a deprecation window because the request wire types carry deny_unknown_fields (DQ-S4).

Relationship to existing anchors

• Add idempotent shell-driven event ingress over the native message bus #137: the ingress acceptance criteria are satisfied by this design (evidence posted there earlier); the command-less stream is its external-producer boundary, and st2-supervised adapters natively cover its bootstrap-timer case.
• Add optional idempotent message ingress #138: superseded by main's --idempotency-key + durable sender ledger (details in the Add idempotent shell-driven event ingress over the native message bus #137 comment); disposition stays with its author.
• Root spec DQ1 (scheduled work): untouched — every remains rejected; a future declared cadence would extend restart policy, not add an in-process loop.
• Topic/thread projection over the native message bus? #49 (topic), Deliver bounded inbox bodies for one-inference DING handling #238 (bounded bodies), Add optional turn-boundary + fresh-idle DING without harness parsing in core #111 (turn-boundary delivery): adjacent, tracked in 04-stream/open-questions.md (DQ-S7 for Deliver bounded inbox bodies for one-inference DING handling #238; the event-side key axis is deliberately not Topic/thread projection over the native message bus? #49's message-side topic).

Open questions (spec DQ-S1..S7)

Producer-identity from grammar for nested streams; stream-state path; ring bound K (measure, don't guess); request-absorption staging; top-level shared streams (defined as a generalization: a nested stream is a top-level stream whose owner and sole recipient is the enclosing agent); parked-stream owner notification; event body bounds vs #238.

Acceptance

Per DQ1's process bar: the executable evidence exists (four experiment records with named tests, all invariant suites green, pre-existing failure sets verified against baseline); upstream review requested — especially on decision 0004 (record kind) and 0005 (locality + vocabulary) before implementation lands.

Posted on behalf of @schickling

field	value
agent_tool	Claude Code
agent_model	claude-fable-5
worktree	st2/schickling/2026-08-20-pipes
machine	dev3

[schickling]

Design addition from a real fleet scenario (an agent burning inference turns re-running pty peek to watch a deploy): waits are modeled as standing keyed feeds, never per-wait subscriptions — stream "pty" { command "pty-lifecycle-watch" } declared once, with each build/deploy as a key emitting superseding phase events plus a terminal exit event. No new mechanism; the alternative (a oneshot completion state) was considered and deferred to DQ-S8 with the finding that the task model has no run-to-completion lifecycle (TaskLifecycle is Service | AdoptOnly), so completion semantics belong there if ever needed, adjacent to DQ1's scheduled exec work. Spec doctrine section and DQ-S8 landed in 017d82e on the VRS PR; pty-lifecycle-watch joins gh-ci-watch as the first two real adapters in the implementation map.