Skip to content

test(agent-spec): prove declared event streams - #79

Merged
schickling merged 17 commits into
mainfrom
schickling/2026-08-20-2026-08-20-stream-e2e
Aug 20, 2026
Merged

test(agent-spec): prove declared event streams#79
schickling merged 17 commits into
mainfrom
schickling/2026-08-20-2026-08-20-stream-e2e

Conversation

@schickling

@schickling schickling commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Problem

st2 #300 implements first-class declared event streams, but the canonical Agent Spec and maintained cross-boundary proof surface still described the older runner. Repository integration tests alone also did not prove that real waiters can remain supervised across a long operation, survive uncertain event acknowledgements, publish one truthful terminal edge, and wake their live owner.

Goal

Make stream declarations part of the canonical Agent Spec and continuously prove both the core contract and representative GitHub CI/Nix waiter use cases through failure-capable E2E cells.

Decisions

  • Pin the corpus to exact st2 source ffdb83c9541978a96ff8ce4c466628e15918cbc1 and the independently reproduced release-mode binary digest adbd2099db237c17df3dac29052cb387f4ed99888e7477910c33e518c377a3e8.
  • Preserve the older immutable published release provenance separately; this exact candidate is not described as a published release.
  • Keep GitHub network state outside the maintained gate. Deterministic controls exercise the same gh --json and event boundaries offline; an explicit authenticated lane supplies live pending-to-terminal evidence.
  • Use actual uncached local Nix derivations for the Nix waiter. One succeeds and one intentionally fails, and both delay long enough for the running waiter state to be observed.
  • Retry identical immutable event arguments after injected transient publication failures. Restarted adapters derive the same event identity from the stable request, so replay must deduplicate rather than create another inbox record or DING delivery.

Verification

  • Final binary reproduction:
    • cargo build --release --locked
    • version identifies source ffdb83c
    • SHA-256: adbd2099db237c17df3dac29052cb387f4ed99888e7477910c33e518c377a3e8
  • st2 eval ./cells/agent-spec-streams/ --keep
    • SCORE: 8 PASS / 0 FAIL / 8 gating judges
  • st2 eval ./cells/stream-github-ci-waiter/ --keep
    • SCORE: 6 PASS / 0 FAIL / 6 gating judges
    • proves pending polling, terminal success, stable replay, failure supersession, transient delivery retry, bounded timeout without emission, and cleanup
  • Live authenticated GitHub lane:
    • Nix run 32274732673, attempt 2, exact head 1ca43e397f804277c553b08b4571408e82831bb4
    • 312 recorded observations: first IN_PROGRESS/pending and last SUCCESS/pass
    • job ran from 2026-08-20T10:29:05Z through 10:43:58Z; waiter emitted the successful terminal result and all six gates passed
  • st2 eval ./cells/stream-nix-build-waiter/ --keep
    • SCORE: 6 PASS / 0 FAIL / 6 gating judges
    • both real derivations remained observable as running for at least 1.8 seconds; success and intentional failure emitted truthful terminal records
    • both terminal events visibly reached the live generated DING sidecar exactly once
    • each public message remained structured and readable, was archived, and reread byte-identically; the inbox drained and the archive retained two records
    • two injected publication failures were retried with unchanged arguments; immediate retry and full adapter restart both deduplicated to the original filenames without recreating an inbox record or redelivering DING; cleanup passed
  • Complete bin/check-corpus.sh:
    • 63 maintained cells: 23 model-free and 40 model-backed
    • 233 shell files parse
    • 119 KDL files parse
    • 63 folder evals and 15 nested fleet catalogs load/validate
    • fixture reset, harness, VRS, mutation, PII, catalog freshness, and complete preflight: PASS

Complexity

The two new cells keep service-specific adapters as shell scripts behind the public st2/gh/Nix interfaces. Deterministic boundary tools exist only inside the GitHub cell; Nix itself and the st2 runtime are never mocked. The Nix DING proof uses a deterministic PTY fixture to exercise the generated sidecar and observable terminal delivery. The added retry loops are bounded and make uncertain delivery behavior explicit.

Concerns

Friction & bottlenecks

  • The evals repository relies on the pinned st2 development environment for Rust/Cargo during the complete preflight; invoking the script from an ambient shell stops at the parser gate when Cargo is absent.
  • The first Nix package build attempt again hit the existing nonblocking runtime-owner-lock test flake. A clean retry built the identical derivation; the release-mode binary, all three focused cells, and the complete corpus gate then passed.
  • The live Nix workflow took about 15 minutes. This is captured as opt-in evidence rather than imposed on routine corpus validation.

Follow-ups

  • Publish and pin immutable release provenance after the st2 stream stack lands.

References

@schickling
schickling marked this pull request as ready for review August 20, 2026 09:26

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 100ad68e3a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread bin/check-corpus.sh Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6e2e86e8cb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread cells/agent-spec-streams/fixture/net/agents/stream/worker/agent.kdl Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7b32359378

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread cells/agent-spec-streams/fixture/contract.sh
Comment thread cells/agent-spec-streams/fixture/net/agents/stream/worker/agent.kdl Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fe05c6d2df

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread cells/agent-spec-streams/fixture/net/agents/stream/worker/agent.kdl Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ab65e64c51

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread cells/stream-github-ci-waiter/fixture/wait-gh-pr-ci.sh

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 93ed7ebc0f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread cells/stream-nix-build-waiter/fixture/net/agents/stream/nix-watcher/agent.kdl Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 572c0859e9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread bin/check-corpus.sh Outdated
Comment thread cells/stream-github-ci-waiter/fixture/wait-gh-pr-ci.sh Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f857869de8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread cells/stream-github-ci-waiter/fixture/contract.sh Outdated
Comment thread cells/stream-nix-build-waiter/fixture/net/fake-tui.pl Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0b5ae32d14

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread cells/stream-nix-build-waiter/fixture/nix-build-waiter.sh Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cdc530a088

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread cells/stream-github-ci-waiter/fixture/wait-gh-pr-ci.sh

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1bcccd83af

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread cells/stream-nix-build-waiter/fixture/nix-build-waiter.sh
@schickling
schickling merged commit bb91a15 into main Aug 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant