Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 65 additions & 3 deletions AGENT-SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ by evals. st2 is the current implementation, not the owner of the contract; a fu
implementation can target the same contract and proofs.

The current corpus proof is pinned to st2
[`0fed14bb5653b67e1d64f1199e240c4c5c612bf7`](https://github.com/compoundingtech/st2/commit/0fed14bb5653b67e1d64f1199e240c4c5c612bf7)
(`0.1.0`, source `0fed14b`). The pin identifies the implementation and version the corpus currently proves; it
[`ffdb83c9541978a96ff8ce4c466628e15918cbc1`](https://github.com/compoundingtech/st2/commit/ffdb83c9541978a96ff8ce4c466628e15918cbc1)
(`0.1.0`, source `ffdb83c`). The pin identifies the implementation and version the corpus currently proves; it
does not transfer ownership of the specification to st2. A proposed behavior change must update this contract
and its maintained proof cells before an implementation claims conformance. Do not infer additional fields or
commands from older corpus fixtures.
Expand Down Expand Up @@ -64,6 +64,10 @@ agent "<identity>" {

resource "work" uri="github-issue://example/project/123"

stream "webhook" {}
stream "ci" { command #"exec ci-adapter --emit-to "$ST_AGENT""# }
stream "deploy" { argv "deploy-adapter" "--json" }

restart {
attempts 3
interval "60s"
Expand Down Expand Up @@ -102,6 +106,9 @@ Supported agent children are:
| `retired #true` | Legacy read-compatible spelling of retirement. New lifecycle transitions use `desired-state` with a rationale. |
| `keep #true` | Freeze dead evidence and suppress collection/restart for every task; retirement still stops live tasks. |
| `resource "name" uri="absolute-uri"` | Binds one uniquely named, externally identified Resource as declaration metadata. |
| `stream "name" {}` | Declares a named external event ingress endpoint. |
| `stream "name" { command "..." }` | Declares an ingress endpoint and a runner-supervised shell adapter. |
| `stream "name" { argv "program" "arg"... }` | Declares an ingress endpoint and a runner-supervised direct-argv adapter. |
| `restart { … }` | Optional service restart policy. |
| `env { KEY "value" }` | Environment inherited by the compact agent task and sidecars. |
| `command "…"` | Compact interactive task named `agent`. |
Expand All @@ -124,6 +131,57 @@ the window reset; `mode "fail"` parks the task after attempts are exhausted and
crash-loop message to `supervisor`. Invalid restart subfields currently fall back to defaults; authors must not
rely on that permissiveness.

<a id="agent-spec-event-streams"></a>

## Event streams

An agent may declare zero or more named streams. A stream name is 1-40 ASCII characters, contains only
lowercase letters, digits, and hyphens, and neither begins nor ends with a hyphen. Names are unique within one
agent. The stream envelope is closed: the node has exactly one
positional name, no properties, and zero or one launch child. A launch is either exactly one positional
`command` string or a non-empty positional-string `argv`; declaring both, extra arguments or properties,
typed nodes, child blocks, unknown children, and `every` all fail validation. Scheduling remains the separate,
reserved `schedule` contract; streams are long-running event sources.

A launch-bearing stream lowers directly to a derived, terminal-free exec task named `stream-<name>` with
runtime id `<host>.<identity>.stream-<name>`. `command` runs through `sh -c`; `argv` executes element zero
directly with the remaining elements unchanged. The derived task inherits agent environment, cwd, restart
policy, suspension, resumption, retirement, and cleanup. An explicit task named `stream-<name>` is therefore a
collision and fails validation. A stream adapter is a sibling companion: it does not make an otherwise
unrunnable agent runnable, and its failure does not relaunch a healthy owning agent.

A commandless stream declares external ingress and creates no task. Neither form is a delivery transport:
streams publish ordinary durable inbox records but do not wake an agent unless it separately declares a
delivery transport such as `ding`. Producers publish with a stable identity:

```sh
st2 event emit <host>.<identity> \
--stream <name> --event-id <producer-id> \
[--key <group>] [--supersede] [--subject <one-line-summary>] \
--message <body> --host <host> --json
```

The recipient must be active and must declare the stream. A first identity/content pair creates one canonical
inbox record; an identical replay returns the same filename as deduplicated; reuse with different content
fails closed. `--supersede` archives the unread predecessor for the same key, or the stream-wide predecessor
without a key, before publishing its successor. Per-stream state is durable and bounded to the latest 128
identities; an identity older than that honest horizon is new again. Event ids, keys, stream names, and subjects
reject control characters before any write.

Ingress and stream authoring resolve their targets through strict catalog discovery. Ingress retains no-follow
capabilities for stream state, its create-new temporary, inbox, and archive operations. An unobservable
declaration entry or a symlinked state/inbox path therefore fails closed and cannot redirect event bytes or
authored declarations outside the owning agent directory.

Executable evidence: [`agent-spec-streams`](cells/agent-spec-streams/) proves strict declaration failures,
direct lowering and runtime identity, command/argv execution, external ingress without a task, stable
deduplication and conflicting-reuse refusal, and suspend/resume coupling through the real st2 CLI and runtime.
[`stream-nix-build-waiter`](cells/stream-nix-build-waiter/) proves supervised adapters against real successful
and failed Nix builds, transient publication retry, restart replay, and deduplication. The maintained offline
lane of [`stream-github-ci-waiter`](cells/stream-github-ci-waiter/) proves pending-to-terminal polling, keyed
supersession, publication retry, timeout refusal, and cleanup; its opt-in authenticated lane exercises the same
boundary against a real GitHub Actions transition without making network state a merge gate.

<a id="agent-spec-resource-bindings"></a>

## Resource bindings
Expand Down Expand Up @@ -542,7 +600,11 @@ Inspect the declaration, every referenced template, and every workspace destinat
materialization command. Materialization is byte-idempotent and does not imply hook installation. Starting
the network is a separate, explicitly authorized action.

For source `0fed14b`, the accepted published Linux executable has SHA256
The latest independently pinned release-mode executable for source `ffdb83c` has SHA256
`adbd2099db237c17df3dac29052cb387f4ed99888e7477910c33e518c377a3e8`. It is built from the exact source
above and exercised by the maintained stream cell. It is not yet an immutable published release artifact.

For the older source `0fed14b`, the accepted published Linux executable has SHA256
`d61d12b2b1189a391c196ca28f8f4ba69072d14fcbad2571fc29db1f250f4eed`; its published archive has SHA256
`d14404ae678bbe3f2a5ad8580cde1e4b8f6009067c46555f392c6e0957b8a2da`, and the downloaded `SHA256SUMS`
asset has SHA256 `50cfd8722e58d1c74fdc543f3e3bb3bac768decd04575fde2360ea838ec5e9d3`. The immutable
Expand Down
10 changes: 8 additions & 2 deletions CATALOG.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
<!-- Generated by bin/generate-catalog.sh from cell KDL and evidence/*.tsv. Do not edit. -->
# Current eval catalog and evidence

This is the sole current corpus surface: **60 included cells** (20 model-free,
This is the sole current corpus surface: **63 included cells** (23 model-free,
40 model-backed) and **4 retired exclusions**. Inventory fields are derived from the active
KDL and executable judge scripts. Latest accepted PASS evidence remains distinct from the append-only run
history: a failed last run is visible without being advertised as accepted evidence, and cells with no
Expand All @@ -18,6 +18,7 @@ harness-native loader plus canonical hook file.
|---|---|---|---:|---|---|---:|---|---|
| `adopt-only-migration` | model-free | — | 0 | none | `90s` | 6 | — | **NO STRUCTURED RUN** |
| `agent-spec-resource-bindings` | model-free | — | 0 | none | `60s` | 4 | — | **NO STRUCTURED RUN** |
| `agent-spec-streams` | model-free | — | 0 | none | `90s` | 7 | — | **NO STRUCTURED RUN** |
| `assignment-contract-cold-assignment` | Codex | `gpt-5.6-sol` / medium | 2 | medium | `1200s` | 3 | — | **NO STRUCTURED RUN** |
| `assignment-contract-cold-focus` | Codex | `gpt-5.6-sol` / medium | 2 | medium | `1200s` | 3 | — | **NO STRUCTURED RUN** |
| `assignment-contract-cold-resources` | Codex | `gpt-5.6-sol` / medium | 2 | medium | `1200s` | 3 | — | **NO STRUCTURED RUN** |
Expand Down Expand Up @@ -63,6 +64,8 @@ harness-native loader plus canonical hook file.
| `skill-inheritance` | Claude | `claude-sonnet-5` / medium | 1 | low | `420s` | 3 | — | **NO STRUCTURED RUN** |
| `st2-doctor-structure` | model-free | — | 0 | none | `60s` | 4 | — | **NO STRUCTURED RUN** |
| `st2-network` | model-free | — | 0 | none | `60s` | 2 | — | **NO STRUCTURED RUN** |
| `stream-github-ci-waiter` | model-free | — | 0 | none | `25m` | 5 | — | **NO STRUCTURED RUN** |
| `stream-nix-build-waiter` | model-free | — | 0 | none | `120s` | 5 | — | **NO STRUCTURED RUN** |
| `strict-validation-json` | model-free | — | 0 | none | `60s` | 4 | — | **NO STRUCTURED RUN** |
| `targeted-reconcile-isolation` | model-free | — | 0 | none | `60s` | 7 | — | **NO STRUCTURED RUN** |
| `test-writing` | Claude | `claude-sonnet-5` / medium | 2 | medium | `1200s` | 3 | — | **NO STRUCTURED RUN** |
Expand Down Expand Up @@ -98,6 +101,7 @@ while the last-run column makes a recorded failure distinct from a cell with no

- `adopt-only-migration` / `cell` — deterministic native adoption/replacement lifecycle probe; no harness seat
- `agent-spec-resource-bindings` / `cell` — deterministic native Agent Spec Resource-envelope probe; no harness seat
- `agent-spec-streams` / `cell` — deterministic native Agent Spec stream lifecycle and ingress probe; no harness seat
- `context-resource-continuity` / `cell` — deterministic native context/resource restart probe; no harness seat
- `docs` / `judge:cold-reader` — one-shot offline Claude print grader; no bus identity, DING, or hook surface
- `exec-lifecycle-logging` / `cell` — deterministic native exec lifecycle probe; no harness seat
Expand All @@ -114,11 +118,13 @@ while the last-run column makes a recorded failure distinct from a cell with no
- `shared-workspace-render-ownership` / `cell` — deterministic native shared-render admission probe; no harness seat
- `st2-doctor-structure` / `cell` — deterministic catalog-health probe; no harness seat
- `st2-network` / `cell` — deterministic catalog/message round-trip probe; no harness seat
- `stream-github-ci-waiter` / `cell` — deterministic GitHub CI waiter controls with an opt-in authenticated read-only smoke; no harness seat
- `stream-nix-build-waiter` / `cell` — deterministic real Nix build waiter and retry probe; no harness seat
- `strict-validation-json` / `cell` — deterministic native validation-diagnostics probe; no harness seat
- `targeted-reconcile-isolation` / `cell` — deterministic selected reconcile isolation probe; no bus-connected agent
- `two-networks-coexist` / `cell` — deterministic catalog-isolation probe; no harness seat

These **21** rows are generated from structured exclusions. A cell-level row is required for
These **24** rows are generated from structured exclusions. A cell-level row is required for
every derived model-free cell. The one-shot docs cold reader is a model-backed grader, but not a bus-connected
agent: it has no st2 identity, DING sidecar, or lifecycle hooks.

Expand Down
8 changes: 5 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,9 +97,11 @@ VERDICT: PASS
```

Requirements are `st2 0.1.0` from source
[`0fed14b`](https://github.com/compoundingtech/st2/commit/0fed14bb5653b67e1d64f1199e240c4c5c612bf7),
`pty`, Bash, Git, `jq`, Rust/Cargo for the pinned KDL parser gate, and Node for JavaScript fixtures. A paid cell
also needs every harness named by its dry-run row.
[`ffdb83c`](https://github.com/compoundingtech/st2/commit/ffdb83c9541978a96ff8ce4c466628e15918cbc1),
`pty`, Bash, Git, `jq`, Rust/Cargo for the pinned KDL parser gate, and Node for JavaScript fixtures. The
`stream-nix-build-waiter` cell additionally requires `nix-build`, `nix-instantiate`, and a configured
`<nixpkgs>` lookup; the overnight runner checks these before starting the selected corpus. A paid cell also
needs every harness named by its dry-run row.

## Cell layout

Expand Down
8 changes: 4 additions & 4 deletions bin/check-corpus.sh
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ set -euo pipefail
repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$repo_root"

expected_source="0fed14b"
expected_binary_sha256="d61d12b2b1189a391c196ca28f8f4ba69072d14fcbad2571fc29db1f250f4eed"
expected_version_regex='^st2 0\.1\.0 — running from local source \(0fed14b, .+ ago\)$'
expected_source="ffdb83c"
expected_binary_sha256="adbd2099db237c17df3dac29052cb387f4ed99888e7477910c33e518c377a3e8"
expected_version_regex='^st2 0\.1\.0 — running from local source \(ffdb83c, .+ ago\)$'
st2_path="$(command -v st2)"
actual_version="$(st2 --version)"
[[ "$actual_version" =~ $expected_version_regex ]] || {
Expand All @@ -19,7 +19,7 @@ actual_binary_sha256="$(sha256sum "$st2_path" | awk '{ print $1 }')"
echo "FAIL: expected st2 binary sha256 $expected_binary_sha256, found $actual_binary_sha256 at $st2_path" >&2
exit 1
}
echo "PASS: pinned published runner source $expected_source ($actual_version; sha256 $actual_binary_sha256)"
echo "PASS: pinned runner source $expected_source ($actual_version; sha256 $actual_binary_sha256)"

mapfile -d '' shell_files < <(
find bin cells -type f -name '*.sh' -not -path '*/_git/*' -print0 | sort -z
Expand Down
19 changes: 18 additions & 1 deletion bin/overnight.sh
Original file line number Diff line number Diff line change
Expand Up @@ -128,9 +128,11 @@ fi

requires_claude=0
requires_codex=0
while IFS=$'\t' read -r _cell _harness models _effort _seats _cost _timeout _judges; do
requires_nix=0
while IFS=$'\t' read -r cell _harness models _effort _seats _cost _timeout _judges; do
[[ "$models" != *claude-sonnet-5* ]] || requires_claude=1
[[ "$models" != *gpt-5.6-sol* ]] || requires_codex=1
[ "$cell" != stream-nix-build-waiter ] || requires_nix=1
done < "$inventory"

printf '%-30s %-10s %-35s %-7s %-5s %-8s %s\n' \
Expand Down Expand Up @@ -182,6 +184,21 @@ echo
echo "== free preflight (no model seats) =="
bin/check-corpus.sh

if [ "$requires_nix" -eq 1 ]; then
command -v nix-build >/dev/null || {
echo "FAIL: selected cells require Nix, but nix-build is not on PATH" >&2
exit 1
}
command -v nix-instantiate >/dev/null || {
echo "FAIL: selected cells require Nix, but nix-instantiate is not on PATH" >&2
exit 1
}
nix-instantiate --find-file nixpkgs >/dev/null 2>&1 || {
echo "FAIL: selected cells require Nix, but <nixpkgs> is not configured" >&2
exit 1
}
fi

if [ "$requires_claude" -eq 1 ]; then
command -v claude >/dev/null || {
echo "FAIL: selected cells require Claude, but claude is not on PATH" >&2
Expand Down
11 changes: 11 additions & 0 deletions cells/agent-spec-streams/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# agent-spec-streams

Model-free E2E acceptance for the canonical Agent Spec stream contract introduced by
[`compoundingtech/st2#300`](https://github.com/compoundingtech/st2/pull/300).

The cell crosses the public parser, reconciliation, process, event-ingress, authoring, inbox, and task
inventory boundaries. Its negative controls prove the contract is fail-closed for invalid names, unsupported
intervals, ambiguous launch shapes, task collisions, undeclared streams, suspended recipients, and conflicting
event identity reuse. It also proves keyed-head to keyless supersession, strict discovery for ingress and
authoring, exact argv byte boundaries, and retained no-follow state/inbox/archive/temp-file capabilities that
prevent symlink redirection outside the agent directory. No model seat or paid provider is started.
33 changes: 33 additions & 0 deletions cells/agent-spec-streams/agent-spec-streams.kdl
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
// Native Agent Spec stream declaration, lifecycle, adapter, and ingress acceptance.
eval {
copy "./fixture"
max-timeout "90s"

run "contract" {
command "bash ./contract.sh"
}

judges {
judge "STRICT SHAPE - names, launch envelope, interval, and task collisions fail closed" {
exec "grep -Fqx STREAM-STRICT-SHAPE-GREEN-83a7 $RUNS_DIR/contract.out"
}
judge "LOWERING - shell and argv adapters have exact derived exec task identities; external ingress has none" {
exec "grep -Fqx STREAM-LOWERING-GREEN-83a7 $RUNS_DIR/contract.out"
}
judge "ADAPTERS - both supervised launch forms publish through the public event CLI" {
exec "grep -Fqx STREAM-ADAPTERS-GREEN-83a7 $RUNS_DIR/contract.out"
}
judge "INGRESS - external events deduplicate stably and conflicting identity reuse fails closed" {
exec "grep -Fqx STREAM-INGRESS-GREEN-83a7 $RUNS_DIR/contract.out"
}
judge "CAPABILITIES - strict discovery and no-follow state/inbox/archive operations prevent redirected writes" {
exec "grep -Fqx STREAM-CAPABILITIES-GREEN-83a7 $RUNS_DIR/contract.out"
}
judge "LIFECYCLE - suspend removes owner plus adapters and refuses ingress; resume relaunches them" {
exec "grep -Fqx STREAM-LIFECYCLE-GREEN-83a7 $RUNS_DIR/contract.out"
}
judge "CLEANUP - no eval-owned PTY or exec task remains" {
exec "grep -Fqx STREAM-CLEANUP-GREEN-83a7 $RUNS_DIR/contract.out"
}
}
}
18 changes: 18 additions & 0 deletions cells/agent-spec-streams/fixture/adapter.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
#!/usr/bin/env bash
set -euo pipefail

stream="${1:?stream name required}"
if test "$stream" = argv; then
test "$#" -eq 3
test "$2" = 'value with spaces'
test "$3" = 'literal;$(not-expanded)'
fi
marker="${CATALOG:?CATALOG required}/../adapter-$stream.json"
st2 event emit "${ST_AGENT:?ST_AGENT required}" \
--stream "$stream" \
--event-id "$stream-delivery-1" \
--subject "$stream adapter" \
--message "payload from $stream" \
--host stream \
--json >"$marker"
exec sleep 300
Loading