Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions INVARIANTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,4 +35,5 @@ materialization, messaging, DING, or presence must preserve them.
| **Tracked workspaces fail closed** | Materialization simulates content operations before writing and refuses a real change to any Git-tracked target. Byte-identical tracked, untracked, and non-Git targets retain useful behavior. | `tests/materialize.rs::every_content_directive_refuses_to_change_a_tracked_target_before_any_write`; `tests/materialize.rs::byte_identical_tracked_target_is_allowed_without_modification`; `tests/materialize.rs::untracked_and_non_git_targets_remain_materializable` |
| **Native flat root** | Without an authored override, catalog tasks, eval messaging, shell helpers, and DING all use the catalog itself as `ST_ROOT`; no nested bus directory is synthesized. | `src/eval_run.rs::bus_root_expands_st_root_else_defaults`; `tests/eval_run_e2e.rs::st2_eval_runs_a_benign_folder_to_a_pass_verdict`; `tests/pty.rs` |
| **Resource observation is state-first, atomic, and fenced** | ABI-3 periodic publication and demanded `Published` results reuse one bounded `Publication` payload and one host acceptance, digest, relevance, typed-fact, and catch-up core; the host never trusts a runtime digest or observation timestamp. Demand reaches only a resident runtime that explicitly declares `capability "demand"`. Every `Observe` carries a positive watermark and the exact owner, binding, and registration, and exactly one matching `Unchanged`, `Failed`, or `Published` atomic result closes it. One outstanding dispatch plus one latest trailing watermark coalesces bursts without losing in-flight arrivals. Backpressure retains queued demand, replacement fences stale output, restart and provider failure settle honestly, and client disconnect or wait expiry never cancels accepted work. | `tests/resource_profile_supervisor_e2e.rs::demand_observation_settlement_matrix_is_atomic_and_preserves_facts`; `tests/resource_profile_supervisor_e2e.rs::demand_observation_coalesces_and_fences_watermarks`; `tests/resource_profile_supervisor_e2e.rs::demand_observation_survives_restart_disconnect_and_denies_missing_capability`; `tests/resource_profile_supervisor_e2e.rs::observable_publication_reaches_builtin_resync_with_filter_catch_up_and_scope_isolation`; `tests/agent_resource.rs::refresh_cli_reports_exact_receipts_and_wait_expiry_keeps_the_request`; `src/resource_observe.rs::tests::receipt_evidence_shape_matches_atomic_results` |
| **Atomic resource proposal publication** | Every changed resource publication is one host-owned compare-and-swap fenced by binding generation, state revision, and prior carrier digest. A persistent cross-process lock admits at most one proposal from the same prior. The content-derived proposal ID binds the accepted carrier digest and semantic outbox envelope; the durable intent becomes eligible only with the exact canonical carrier, then folds into one authoritative catch-up state. A pre-carrier crash exposes old state, a post-carrier crash catches up on restart, and retry after a lost acknowledgement returns the durable receipt without another transition. Ordinary reconciliation fails closed on out-of-band divergence after a committed intent; only an explicit generation-advance recovery may re-adopt the canonical carrier or its absence while invalidating the old intent and fence. | `src/resource_profile.rs::tests::atomic_publication_fences_races_and_survives_crash_restarts`; `src/resource_profile.rs::tests::generation_advance_explicitly_recovers_diverged_or_missing_carrier` |
| **Proof references resolve** | Every qualified test named in this table exists in its named source file, so stale invariant claims fail the suite instead of silently surviving a refactor. | `tests/invariants.rs::qualified_proof_references_resolve` |
151 changes: 151 additions & 0 deletions crates/st2-resource-protocol/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -543,6 +543,130 @@ impl<'de> Deserialize<'de> for SnapshotDigest {
}
}

/// Content-derived identity of one host-validated publication proposal.
///
/// The host derives this after validating the publication. It is deliberately distinct from the
/// snapshot digest: the same bytes proposed against a different binding, generation, revision,
/// prior digest, selected-topic set, or ordered fact envelope are a different proposal.
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)]
#[serde(transparent)]
pub struct ProposalId(SnapshotDigest);

impl ProposalId {
pub fn of(bytes: &[u8]) -> Self {
Self(SnapshotDigest::of(bytes))
}

pub fn as_bytes(&self) -> &[u8; 32] {
self.0.as_bytes()
}
}

impl fmt::Display for ProposalId {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
fmt::Display::fmt(&self.0, formatter)
}
}

/// Compare-and-swap fence captured before a provider computes a publication.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "camelCase", deny_unknown_fields)]
pub struct ProposalFence {
generation: u64,
revision: u64,
#[serde(skip_serializing_if = "Option::is_none")]
prior_digest: Option<SnapshotDigest>,
}

impl ProposalFence {
pub fn new(
generation: u64,
revision: u64,
prior_digest: Option<SnapshotDigest>,
) -> Self {
Self {
generation,
revision,
prior_digest,
}
}

pub fn generation(&self) -> u64 {
self.generation
}

pub fn revision(&self) -> u64 {
self.revision
}

pub fn prior_digest(&self) -> Option<SnapshotDigest> {
self.prior_digest
}
}

/// Durable receipt for one changed publication.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "camelCase", deny_unknown_fields)]
pub struct PublicationCommit {
proposal_id: ProposalId,
generation: u64,
revision: u64,
digest: SnapshotDigest,
}

impl PublicationCommit {
pub fn new(
proposal_id: ProposalId,
generation: u64,
revision: u64,
digest: SnapshotDigest,
) -> Self {
Self {
proposal_id,
generation,
revision,
digest,
}
}

pub fn proposal_id(&self) -> ProposalId {
self.proposal_id
}

pub fn generation(&self) -> u64 {
self.generation
}

pub fn revision(&self) -> u64 {
self.revision
}

pub fn digest(&self) -> SnapshotDigest {
self.digest
}
}

/// Result of the host's one authoritative compare-and-swap transition.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ProposalCommit {
Committed(PublicationCommit),
AlreadyCommitted(PublicationCommit),
Unchanged {
generation: u64,
revision: u64,
digest: SnapshotDigest,
},
StaleGeneration {
actual_generation: u64,
actual_revision: u64,
},
StalePrior {
actual_generation: u64,
actual_revision: u64,
actual_digest: Option<SnapshotDigest>,
},
}

#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
#[serde(
tag = "type",
Expand Down Expand Up @@ -993,6 +1117,33 @@ mod tests {
}
}

#[test]
fn proposal_fence_and_commit_receipt_are_domain_typed() {
let prior = SnapshotDigest::of(b"prior");
let fence = ProposalFence::new(7, 11, Some(prior));
assert_eq!(
serde_json::to_value(fence).unwrap(),
json!({
"generation": 7,
"revision": 11,
"priorDigest": prior.to_string(),
})
);
assert_eq!(
serde_json::from_value::<ProposalFence>(serde_json::to_value(fence).unwrap()).unwrap(),
fence
);

let proposal_id = ProposalId::of(b"proposal identity");
let digest = SnapshotDigest::of(b"carrier");
let commit = PublicationCommit::new(proposal_id, 7, 12, digest);
assert_eq!(commit.proposal_id(), proposal_id);
assert_eq!(commit.generation(), 7);
assert_eq!(commit.revision(), 12);
assert_eq!(commit.digest(), digest);
assert_ne!(proposal_id.as_bytes(), digest.as_bytes());
}

#[test]
fn host_frames_have_exact_json_shape_and_newline() {
let register = HostMessage::Register {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -126,3 +126,96 @@ unprojected inputs, and load a relative module from the applied live catalog.
missing newly introduced module.
- Resolver observability and same-path module-cache invalidation remain explicit
design questions; neither weakens the containment and feature-gating contract.

## Amendment — 2026-09-01 (Q39)

Johannes approved Q39 to make WASIp2 Component Model components the universal
execution envelope for observable Resource providers. This amendment preserves
the original decision's closed core-wasm resolver and replaces only the
observable host-process mechanism described later by
[decision 0014](./0014-resource-profiles-are-state-first-read-and-observe-capabilities.md).
Decision 0014's state-first authority, demand semantics, typed `Publication`,
semantic filtering, and catch-up model remain in force.

### Context

The closed resolver has one pure job: map an opaque Resource URI to a contained
carrier path. It needs neither provider I/O nor the Component Model. Observable
providers have a different job: call a remote or local provider, normalize its
domain state, and propose a canonical publication. A catalog-trusted native
process can perform that job, but it carries ambient host authority and creates
a second long-lived lifecycle and JSON protocol beside Wasmtime.

Five disposable prototypes tested a narrower boundary with Wasmtime 48.0.1.
Typed GitHub and PTY observations proved real domain I/O without exposing raw
HTTP, caller-selected executable/arguments, environment, filesystem, or socket
access. Fresh-Store cancellation tests ended with zero active tasks or
capabilities. Verified compiled-code reuse kept a small provider's AOT disk-hit
p50 at 0.482 ms and fresh Store plus instance observation p50 at 47.751 µs. An
independent multiprocess oracle passed three 30-process runs covering
one-winner compare-and-swap, stale generation, process-crash boundaries,
acknowledgement loss, deterministic outbox identity, and restart catch-up.

### Decision

1. The core-wasm resolver ABI, its no-import sandbox, fresh resolution Store,
path containment, registry behavior, feature gate, and transactional module
ownership remain the only resolution mechanism.
2. Every observable provider executes through one versioned WASIp2 Component
Model envelope. st2 does not maintain a parallel native or host-process
provider framework.
3. A provider component may import only explicit provider-domain capabilities
linked by the host. The host owns credentials, allowlists, limits,
deadlines, cancellation, and redacted typed failures. No ambient WASI
command, environment, clock, random, process, raw HTTP, filesystem, or
socket authority is linked.
4. Every descriptor call and observation receives a fresh Store and component
instance. Engine, Linker, compiled Component, and compatible host-produced
AOT bytes may be reused; Store and instance state may not be pooled or reset.
5. The component returns `Unchanged`, a typed failure, or the existing
`Publication` payload. It never writes the carrier, state record, receipt, or
outbox. The host pairs a publication with
`ProposalFence { generation, revision, prior_digest }`, validates it, and
owns the only atomic commit.
6. One durable transition makes the carrier, resulting digest and revision,
freshness and catch-up state, and deterministic `PublicationIntent` visible
together. A crash before publication leaves the old state. A crash after
publication but before acknowledgement leaves the intent retryable;
delivery remains separate and idempotent.
7. WASIp3 production execution, Store pooling, generic exec/raw
HTTP/filesystem/socket authority, a parallel native provider framework, and
runtime WAC graphs are explicit non-goals. Each requires new evidence and a
further accepted decision rather than an alternate dormant path.

### Evidence and argument

The durable record is the
[WASIp2 component and atomic publication experiment](../07-resource-profile/.experiments/2026-09-01-wasip2-component-and-atomic-publication-prototypes.md).
The disposable harnesses prove the boundary; their local source, result, cache,
and state paths are not production interfaces or scaffolding.

The Component Model is selected for observable providers because its typed
imports make authority reviewable at link time and its typed export preserves
one observation result. Fresh Stores remove the need for an incomplete reset
protocol across guest memory, resources, host state, traps, and cancellation.
Host-only commit prevents a network- or command-capable guest from bypassing
validation or racing publication against settlement. The deterministic durable
outbox makes publication and delivery intent one crash-consistent fact without
claiming exactly-once effects across an external sink.

### Consequences

- A profile may carry two wasm artifacts with deliberately disjoint jobs: a
closed core module for resolution and, only when observable, a WASIp2
component for provider observation.
- Provider support requires a reviewed typed host capability; adding a generic
authority under a domain-flavored name is non-conforming.
- Domain acquisition state such as ETags, cursors, rate limits, and webhook
repair cannot rely on guest Store lifetime. It belongs in bounded host-owned
capability state or explicit durable provider state.
- Compiled-code caching is an optimization, not an authority transfer. Any AOT
deserialize path must authenticate exact host-produced bytes and the complete
engine-compatibility key before crossing Wasmtime's unsafe boundary.
- Decision 0014's host-process topology and newline-delimited JSON mechanism
are superseded. Its publication, demand, delivery, and state-first semantics
apply to direct component invocations.
Original file line number Diff line number Diff line change
Expand Up @@ -65,3 +65,13 @@ Two complexity reductions survive the prototypes:
- Remove `reconcile` and `shutdown` from the normative runtime protocol.
- Specify EOF as runtime termination and supervisor lifecycle as the only shutdown authority.
- Keep restart/backoff policy in existing task lifecycle machinery rather than the profile protocol.

## Subsequent evidence

Q39 retains the selector round-trip and directional-fencing findings as
evidence, but supersedes the executable JSON-line runtime, including its
process-EOF lifecycle, as the observable-provider mechanism. The
[WASIp2 component and atomic publication prototypes](./2026-09-01-wasip2-component-and-atomic-publication-prototypes.md)
proved a narrower universal envelope: one fresh Store and component invocation,
domain-typed host capabilities, and a generation/revision/prior-digest proposal
fence. No production path or protocol depends on the disposable runtime driver.
Original file line number Diff line number Diff line change
Expand Up @@ -61,3 +61,12 @@ The same lifecycle reducer is independent of shared versus per-binding runtime t
- Reuse existing event supersession and DING transport for thin invalidations.
- Keep provider cursors, webhook delivery identifiers, polling intervals, and observation repair inside the profile implementation.
- Specify shared and per-binding runtimes behind one normalized host protocol; do not duplicate delivery state machines.

## Subsequent evidence

Q39 preserves the topology-independent catch-up reducer but supersedes the
shared/per-binding host-process mechanism in the original VRS impact. All
observable providers now use one fresh-Store WASIp2 component invocation.
Provider cursors and repair state remain outside the guest Store in host-owned
domain capability state or explicit durable provider state. The delivery
reducer remains independent of how those typed capabilities acquire input.
Loading
Loading