Add idempotent shell-driven event ingress over the native message bus

[schickling-assistant]

Problem

st2 has a durable native message bus and DING delivery, and st2 message send <recipient> already accepts a body on stdin. That is enough for a one-off shell producer, but not yet a reliable external-event ingress contract.

Recurring timers, GitHub/webhook adapters, monitoring alerts, and similar producers need to turn an external event into exactly one durable inbox item. Today, retrying message send creates a new message each time, and the resulting message has no first-class source event identity that can serve as a receipt and correlation key.

A concrete motivating case is a Nix-managed daily operational check: a systemd timer should emit one event per local calendar day to a standing agent; the inbox write should wake the agent through the existing DING path. The same primitive should work for GitHub delivery IDs and monitoring events without teaching st2 about cron or GitHub.

Proposed ownership boundary

Bootstrap external producers may own when and how an event occurs:

• systemd/launchd/cron timers (bootstrap until native recurring sources exist)
• GitHub webhook or polling adapters
• monitoring and incident sources
• arbitrary shell pipelines

st2 owns the durable ingress boundary and the minimal recurring-source lifecycle:

• declare and supervise a recurring shell-driven source so bootstrap host timers can be removed

• parse a small event envelope from CLI flags and/or stdin

• route it to an agent inbox

• deduplicate retries by (source, event-id)

• return a stable machine-readable receipt

• let the existing DING sidecar wake the recipient

This could be exposed as st2 event emit, st2 pipe, or an extension of st2 message send; the command name is intentionally open.

Minimum useful interface

Conceptually:

check_dev3 | st2 event emit dev3.janitor \
  --source systemd:dev3-janitor-daily \
  --event-id 2026-07-31 \
  --subject 'Daily operational check' \
  --json

A GitHub adapter could use github:<delivery-id> as the source/event identity. Repeating the command with the same identity should return the original receipt and must not create or DING a second inbox item.

The durable inbox item should retain enough provenance to answer:

• which producer emitted it?
• what external event ID caused it?
• which st2 message/receipt represents it?
• was this invocation newly accepted or deduplicated?

Acceptance criteria

• A shell command can emit an event to a catalog identity using stdin for the payload.
• A native recurring shell-driven source can emit on a declared cadence with persistent catch-up semantics, allowing a bootstrap system timer to be removed.
• The producer supplies a stable source and event ID.
• Replaying the same (source, event-id) is idempotent, including concurrent retries.
• JSON output includes at least recipient, message/receipt ID, and created versus deduplicated outcome.
• A newly accepted event follows the normal inbox -> DING path; a duplicate does not cause another DING.
• The on-disk representation remains compatible with normal message list/read/archive/thread workflows, or the CLI clearly defines the join between an event receipt and its message.
• Tests cover retry, concurrent duplicate emission, and a producer crash/retry boundary.

Non-goals

• A general-purpose cron or workflow engine. The minimal recurring event source above is in scope.
• Embedding GitHub polling/webhook logic in st2.
• A general workflow engine.
• Replacing the native message bus or DING transport.

Open design questions

• Extend message frontmatter with source / event-id, or keep a separate receipt index joined to the immutable message?
• Should deduplication live per recipient/catalog, or at a shared bus scope?
• What retention rule applies to dedupe receipts after the inbox message is archived?
• Is event emit a clearer user model than pipe, given that stdin piping already works for ordinary messages?

Posted on behalf of @schickling

field	value
agent_name	co3-cobalt
agent_session_id	b9db6156-4852-4586-87da-29efe25f759f
agent_tool	Codex CLI
agent_tool_version	0.145.0
agent_runtime	Codex CLI 0.145.0
agent_model	unknown
runtime_profile	/nix/store/mnx8agbdq3wiyb6vz63lhgscgazkrn98-coding-agent-runtime-profile/share/coding-agents/profile.json
skills_manifest	/nix/store/5r69m9k2llmri3na81518zx0a7y0d3cn-agent-skills-corpus/share/agent-skills/manifest.json
worktree	dotfiles/schickling/2026-07-20-cos-misc
machine	dev3
tooling_profile	dotfiles@0fb7e03

[myobie]

Nathan likes this direction. Our st2 owner is building a minimal draft PR now. The first version adds optional --source and --event-id flags and frontmatter to ordinary st2 message send. Without those flags, message behavior and output do not change. With both flags, retries on one local catalog file system return the first receipt and create no second inbox item or DING. The proof covers inbox, archive, concurrent retry, and interrupted publication. This does not add a scheduler, GitHub logic, CAS, or workflow policy. We will link the draft PR here when it is ready.

[myobie]

Draft implementation: PR #138.

Exact head: 9fa382389f961868551debafd31cdd2205d92a6a.

Current evidence: focused message tests, message CLI tests, Clippy, formatting, doctests, and hosted Nix run 30664286396 are green. The full workspace/all-target suite passes with three exact exclusions that fail identically on current main; those baseline failures are recorded in the PR body.

The PR remains a draft. Human review is required. This is not an acceptance or merge-ready signal.

[schickling]

Executable evidence toward this issue's contract, from two e2e prototypes run on current main (as of e4e9d56). A dedicated "pipes" design issue with the full VRS proposal will follow; posting the findings that bear on this issue's open questions now.

What was prototyped

1. Ingress: (source, event-id) idempotent emission, implemented two ways end-to-end — (a) ordinary message send with --source/--event-id frontmatter and a derived idempotency key, (b) the existing service-principal request send path. 16 evidence tests: single + 12-way concurrent replay → exactly one inbox item and one DING; conflicting key reuse refused; replay after archive re-notifies in neither path.
2. Source lifecycle: a declared pipe "gh-ci" { command "…" } node lowering to a derived exec companion (mirroring the generated DING sidecar). 27 tests: launch-with-agent in one pass, suspend/retire teardown, crash-loop parks the pipe without disturbing the agent and surfaces to the supervisor over the bus, honest st2 tasks --json. No changes to run.rs, flapping, park, or task inventory; all derived-DING invariant tests stay green.

Findings on this issue's open design questions

• Frontmatter vs separate receipt index: HEAD (since the sender-ledger work) already carries a durable --idempotency-key scoped (sender, recipient, key) with created vs deduplicated outcomes — the retry/receipt half of this issue's acceptance criteria exists today. Source/event-id provenance as message frontmatter also works and reads back through message ls/read --json.
• Dedup scope: a content hash is measurably insufficient — it cannot distinguish replay-after-producer-restart from a genuine repeat (test: emitting ci failed twice legitimately). A producer-supplied event-id is the contract, not a nicety. Key derivation <source>#<event-id> proved out.
• Sender identity is the real fork: the ordinary-message path cannot express a non-agent producer (the sender must resolve to a catalog agent), so an event source either impersonates an agent (renders a peer ← DING glyph, writes Agent Sent rows) or self-sends (renders ↺ "from myself" for an external event). The service-principal path required zero core changes and closes the loop with typed pending|replied receipts, but has fixable presentation gaps (hardcoded request <key> subject, overloaded ? glyph, no fan-out). Which model is the long-term ideal is under active design exploration; this looks like the load-bearing decision for the "command name" question this issue left open.
• DING path: a newly accepted event follows the normal inbox → DING path and a duplicate does not re-DING — proven in both representations. Burst behavior is FIFO one-notice-per-delivery; coalesce-to-latest cannot live in the DING sidecar (it stages exactly one payload and never deletes inbox items), so supersession must be producer-side.

On draft PR #138

Respectfully, the prototype evidence suggests #138 is superseded by what landed on main since it was drafted: --idempotency-key now exists backed by the durable sender key ledger, whereas #138's inbox→archive scan forgets the key once the archived file is deleted (a much-later replay then creates a second inbox item). Happy to leave its disposition to its author — the acceptance criteria it targeted are otherwise covered by the evidence above.

Not yet settled (in design)

The producer identity model (above), catch-up/cursor semantics for recurring sources, supersession policy, and the KDL/CLI surface. These land in the follow-up design issue with the VRS proposal.

Posted on behalf of @schickling

field	value
agent_tool	Claude Code
agent_model	claude-fable-5
worktree	st2/schickling/2026-08-20-pipes
machine	dev3