diff --git a/INVARIANTS.md b/INVARIANTS.md index 8dcd5ad9..8e6a5a2c 100644 --- a/INVARIANTS.md +++ b/INVARIANTS.md @@ -34,4 +34,5 @@ materialization, messaging, DING, or presence must preserve them. | **Parked tasks are visible and individually recoverable** | A parked task is reported alongside an unmodified runtime observation as a complete known fault; only an unbelievable marker fails closed. Park markers and unpark requests share the exact canonical catalog-folder plus host ownership scope, and the projected recovery argv carries both axes, so same-host supervisors cannot see, delete, consume, or advertise recovery into each other's channels even for the same task ID. A projected park whose supervisor generation is gone is positively not parked. An explicit per-task unpark clears that task's park and spent budget so it is launchable again and stays recovered past `interval`, releases no other parked task, restarts no healthy peer, and restores the agent's derived DING. | `src/flapping.rs::unpark_restores_a_launchable_task_not_just_a_cleared_flag`; `src/flapping.rs::unpark_is_per_task_and_reports_whether_it_changed_anything`; `src/park.rs::same_host_supervisors_isolate_markers_and_requests_by_catalog`; `src/park.rs::a_marker_from_a_dead_supervisor_reads_as_not_parked`; `src/park.rs::published_parks_are_readable_and_clear_when_the_task_recovers`; `src/park.rs::an_unbelievable_marker_is_indeterminate_not_absent`; `src/park.rs::a_request_is_consumed_exactly_once`; `src/task_inventory.rs::a_parked_task_reports_its_fault_alongside_a_truthful_runtime_state`; `src/task_inventory.rs::an_unbelievable_park_marker_makes_the_envelope_incomplete`; `tests/task_inventory_cli.rs::projected_recovery_targets_its_exact_catalog_and_host_despite_ambient_defaults`; `tests/run.rs::an_operator_recovers_one_parked_task_without_disturbing_a_healthy_peer`; `tests/run.rs::an_unpark_request_for_a_task_that_is_not_parked_says_so` | | **Tracked workspaces fail closed** | Materialization simulates content operations before writing and refuses a real change to any Git-tracked target. Byte-identical tracked, untracked, and non-Git targets retain useful behavior. | `tests/materialize.rs::every_content_directive_refuses_to_change_a_tracked_target_before_any_write`; `tests/materialize.rs::byte_identical_tracked_target_is_allowed_without_modification`; `tests/materialize.rs::untracked_and_non_git_targets_remain_materializable` | | **Native flat root** | Without an authored override, catalog tasks, eval messaging, shell helpers, and DING all use the catalog itself as `ST_ROOT`; no nested bus directory is synthesized. | `src/eval_run.rs::bus_root_expands_st_root_else_defaults`; `tests/eval_run_e2e.rs::st2_eval_runs_a_benign_folder_to_a_pass_verdict`; `tests/pty.rs` | +| **Resource observation is state-first, atomic, and fenced** | ABI-3 periodic publication and demanded `Published` results reuse one bounded `Publication` payload and one host acceptance, digest, relevance, typed-fact, and catch-up core; the host never trusts a runtime digest or observation timestamp. Demand reaches only a resident runtime that explicitly declares `capability "demand"`. Every `Observe` carries a positive watermark and the exact owner, binding, and registration, and exactly one matching `Unchanged`, `Failed`, or `Published` atomic result closes it. One outstanding dispatch plus one latest trailing watermark coalesces bursts without losing in-flight arrivals. Backpressure retains queued demand, replacement fences stale output, restart and provider failure settle honestly, and client disconnect or wait expiry never cancels accepted work. | `tests/resource_profile_supervisor_e2e.rs::demand_observation_settlement_matrix_is_atomic_and_preserves_facts`; `tests/resource_profile_supervisor_e2e.rs::demand_observation_coalesces_and_fences_watermarks`; `tests/resource_profile_supervisor_e2e.rs::demand_observation_survives_restart_disconnect_and_denies_missing_capability`; `tests/resource_profile_supervisor_e2e.rs::observable_publication_reaches_builtin_resync_with_filter_catch_up_and_scope_isolation`; `tests/agent_resource.rs::refresh_cli_reports_exact_receipts_and_wait_expiry_keeps_the_request`; `src/resource_observe.rs::tests::receipt_evidence_shape_matches_atomic_results` | | **Proof references resolve** | Every qualified test named in this table exists in its named source file, so stale invariant claims fail the suite instead of silently surviving a refactor. | `tests/invariants.rs::qualified_proof_references_resolve` | diff --git a/crates/st2-resource-protocol/src/lib.rs b/crates/st2-resource-protocol/src/lib.rs index f765a61b..05c6920f 100644 --- a/crates/st2-resource-protocol/src/lib.rs +++ b/crates/st2-resource-protocol/src/lib.rs @@ -13,6 +13,7 @@ pub const MAX_PROTOCOL_LINE_BYTES: usize = 2 * 1024 * 1024; pub const MAX_SNAPSHOT_BYTES: usize = 1024 * 1024; pub const MAX_SELECTOR_BYTES: usize = 16 * 1024; pub const MAX_HEALTH_DETAIL_BYTES: usize = 16 * 1024; +pub const MAX_OBSERVATION_DIAGNOSTIC_BYTES: usize = 16 * 1024; const MAX_OPAQUE_ID_BYTES: usize = 16 * 1024; /// Fact bounds are deliberately small relative to the 2 MiB frame: even 32 facts whose strings @@ -95,10 +96,7 @@ impl ResourceFact { Ok(fact) } - pub fn current( - key: impl Into, - value: impl Into, - ) -> Result { + pub fn current(key: impl Into, value: impl Into) -> Result { Self::new(key, FactValue::Omitted, FactValue::value(value)) } @@ -167,7 +165,9 @@ fn validate_fact_string( #[derive(Debug, Clone, PartialEq, Eq)] pub enum FactError { - TooMany { actual: usize }, + TooMany { + actual: usize, + }, Empty { field: &'static str, }, @@ -186,7 +186,10 @@ impl fmt::Display for FactError { fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { match self { Self::TooMany { actual } => { - write!(formatter, "fact list has {actual} entries; maximum is {MAX_FACTS}") + write!( + formatter, + "fact list has {actual} entries; maximum is {MAX_FACTS}" + ) } Self::Empty { field } => write!(formatter, "fact {field} must not be empty"), Self::TooLarge { @@ -200,9 +203,7 @@ impl fmt::Display for FactError { Self::NotPrintable { field } => { write!(formatter, "fact {field} must be one printable line") } - Self::MissingValue => { - formatter.write_str("fact must include before, after, or both") - } + Self::MissingValue => formatter.write_str("fact must include before, after, or both"), } } } @@ -565,6 +566,12 @@ pub enum HostMessage { binding_id: BindingId, registration: RegistrationToken, }, + Observe { + owner: RuntimeOwner, + binding_id: BindingId, + registration: RegistrationToken, + demand_watermark: u64, + }, } #[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] @@ -577,25 +584,79 @@ pub enum RuntimeHealthState { } #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase", deny_unknown_fields)] +pub struct Publication { + pub schema_id: String, + pub media_type: String, + pub bytes: SnapshotBytes, + pub topics: Vec, + #[serde(skip_serializing_if = "Option::is_none")] + pub facts: Option>, +} + +impl Publication { + fn validate(&self) -> Result<(), ProtocolError> { + validate_topics(&self.topics)?; + validate_facts(self.facts.as_deref().unwrap_or_default()) + } +} + +#[derive(Debug, Clone, PartialEq, Serialize)] #[serde( - tag = "type", + tag = "status", + rename_all = "camelCase", + rename_all_fields = "camelCase" +)] +pub enum ObservationResult { + Unchanged, + Failed { + #[serde(skip_serializing_if = "Option::is_none")] + diagnostic: Option, + }, + Published { + publication: Publication, + }, +} + +#[derive(Deserialize)] +#[serde( + tag = "status", rename_all = "camelCase", rename_all_fields = "camelCase", deny_unknown_fields )] +enum ObservationResultWire { + Unchanged {}, + Failed { diagnostic: Option }, + Published { publication: Publication }, +} + +impl<'de> Deserialize<'de> for ObservationResult { + fn deserialize(deserializer: D) -> Result + where + D: Deserializer<'de>, + { + Ok(match ObservationResultWire::deserialize(deserializer)? { + ObservationResultWire::Unchanged {} => Self::Unchanged, + ObservationResultWire::Failed { diagnostic } => Self::Failed { diagnostic }, + ObservationResultWire::Published { publication } => Self::Published { publication }, + }) + } +} + +#[derive(Debug, Clone, PartialEq, Serialize)] +#[serde( + tag = "type", + rename_all = "camelCase", + rename_all_fields = "camelCase" +)] pub enum RuntimeMessage { Publish { owner: RuntimeOwner, binding_id: BindingId, registration: RegistrationToken, - schema_id: String, - media_type: String, - bytes: SnapshotBytes, - topics: Vec, - #[serde(skip_serializing_if = "Option::is_none")] - facts: Option>, - #[serde(skip_serializing_if = "Option::is_none")] - observed_at: Option, + #[serde(flatten)] + publication: Publication, }, Health { owner: RuntimeOwner, @@ -607,6 +668,98 @@ pub enum RuntimeMessage { #[serde(skip_serializing_if = "Option::is_none")] detail: Option, }, + ObservationResult { + owner: RuntimeOwner, + binding_id: BindingId, + registration: RegistrationToken, + demand_watermark: u64, + result: ObservationResult, + }, +} + +#[derive(Deserialize)] +#[serde( + tag = "type", + rename_all = "camelCase", + rename_all_fields = "camelCase", + deny_unknown_fields +)] +enum RuntimeMessageWire { + Publish { + owner: RuntimeOwner, + binding_id: BindingId, + registration: RegistrationToken, + /// ABI-3 runtimes may still send this retired producer timestamp. The host never trusted it, + /// so decode it only to preserve that ABI and discard it at this boundary. + observed_at: Option, + #[serde(flatten)] + publication: Publication, + }, + Health { + owner: RuntimeOwner, + binding_id: Option, + registration: Option, + state: RuntimeHealthState, + detail: Option, + }, + ObservationResult { + owner: RuntimeOwner, + binding_id: BindingId, + registration: RegistrationToken, + demand_watermark: u64, + result: ObservationResult, + }, +} + +impl<'de> Deserialize<'de> for RuntimeMessage { + fn deserialize(deserializer: D) -> Result + where + D: Deserializer<'de>, + { + Ok(match RuntimeMessageWire::deserialize(deserializer)? { + RuntimeMessageWire::Publish { + owner, + binding_id, + registration, + observed_at, + publication, + } => { + drop(observed_at); + Self::Publish { + owner, + binding_id, + registration, + publication, + } + }, + RuntimeMessageWire::Health { + owner, + binding_id, + registration, + state, + detail, + } => Self::Health { + owner, + binding_id, + registration, + state, + detail, + }, + RuntimeMessageWire::ObservationResult { + owner, + binding_id, + registration, + demand_watermark, + result, + } => Self::ObservationResult { + owner, + binding_id, + registration, + demand_watermark, + result, + }, + }) + } } #[derive(Debug)] @@ -617,6 +770,8 @@ pub enum ProtocolError { LineTooLarge { actual: usize }, SelectorTooLarge { actual: usize }, HealthDetailTooLarge { actual: usize }, + ObservationDiagnosticTooLarge { actual: usize }, + InvalidDemandWatermark, InvalidTopics(&'static str), InvalidFacts(FactError), InvalidHealthScope, @@ -641,6 +796,13 @@ impl fmt::Display for ProtocolError { formatter, "health detail is {actual} bytes; maximum is {MAX_HEALTH_DETAIL_BYTES}" ), + Self::ObservationDiagnosticTooLarge { actual } => write!( + formatter, + "observation diagnostic is {actual} bytes; maximum is {MAX_OBSERVATION_DIAGNOSTIC_BYTES}" + ), + Self::InvalidDemandWatermark => { + formatter.write_str("observation demand watermark must be positive") + } Self::InvalidTopics(reason) => write!(formatter, "invalid topics: {reason}"), Self::InvalidFacts(error) => write!(formatter, "invalid facts: {error}"), Self::InvalidHealthScope => formatter @@ -710,23 +872,26 @@ fn encode_protocol_line(message: &impl Serialize) -> Result, ProtocolErr } fn validate_host_message(message: &HostMessage) -> Result<(), ProtocolError> { - if let HostMessage::Register { selector, .. } = message { - let actual = serde_json::to_vec(selector) - .map_err(ProtocolError::Json)? - .len(); - if actual > MAX_SELECTOR_BYTES { - return Err(ProtocolError::SelectorTooLarge { actual }); + match message { + HostMessage::Register { selector, .. } => { + let actual = serde_json::to_vec(selector) + .map_err(ProtocolError::Json)? + .len(); + if actual > MAX_SELECTOR_BYTES { + return Err(ProtocolError::SelectorTooLarge { actual }); + } + Ok(()) } + HostMessage::Observe { + demand_watermark, .. + } if *demand_watermark == 0 => Err(ProtocolError::InvalidDemandWatermark), + HostMessage::Observe { .. } | HostMessage::Unregister { .. } => Ok(()), } - Ok(()) } fn validate_runtime_message(message: &RuntimeMessage) -> Result<(), ProtocolError> { match message { - RuntimeMessage::Publish { topics, facts, .. } => { - validate_topics(topics)?; - validate_facts(facts.as_deref().unwrap_or_default()) - } + RuntimeMessage::Publish { publication, .. } => publication.validate(), RuntimeMessage::Health { binding_id, registration, @@ -745,6 +910,29 @@ fn validate_runtime_message(message: &RuntimeMessage) -> Result<(), ProtocolErro } Ok(()) } + RuntimeMessage::ObservationResult { + demand_watermark, + result, + .. + } => { + if *demand_watermark == 0 { + return Err(ProtocolError::InvalidDemandWatermark); + } + match result { + ObservationResult::Unchanged => Ok(()), + ObservationResult::Failed { diagnostic } => { + if let Some(diagnostic) = diagnostic + && diagnostic.len() > MAX_OBSERVATION_DIAGNOSTIC_BYTES + { + return Err(ProtocolError::ObservationDiagnosticTooLarge { + actual: diagnostic.len(), + }); + } + Ok(()) + } + ObservationResult::Published { publication } => publication.validate(), + } + } } } fn validate_facts(facts: &[ResourceFact]) -> Result<(), ProtocolError> { @@ -786,17 +974,22 @@ mod tests { ) } - fn publish(bytes: &[u8]) -> RuntimeMessage { - RuntimeMessage::Publish { - owner: owner(), - binding_id: BindingId::new("binding").unwrap(), - registration: RegistrationToken::new("registration").unwrap(), + fn publication(bytes: &[u8]) -> Publication { + Publication { schema_id: "schema.v1".to_owned(), media_type: "application/json".to_owned(), bytes: SnapshotBytes::new(bytes.to_vec()).unwrap(), topics: vec!["selected".to_owned()], facts: None, - observed_at: Some("2026-08-30T00:00:00Z".to_owned()), + } + } + + fn publish(bytes: &[u8]) -> RuntimeMessage { + RuntimeMessage::Publish { + owner: owner(), + binding_id: BindingId::new("binding").unwrap(), + registration: RegistrationToken::new("registration").unwrap(), + publication: publication(bytes), } } @@ -824,13 +1017,23 @@ mod tests { encode_host_line(&unregister).unwrap(), b"{\"type\":\"unregister\",\"owner\":{\"incarnation\":\"incarnation\",\"claim\":\"claim\"},\"bindingId\":\"binding\",\"registration\":\"registration\"}\n" ); + let observe = HostMessage::Observe { + owner: owner(), + binding_id: BindingId::new("binding").unwrap(), + registration: RegistrationToken::new("registration").unwrap(), + demand_watermark: 7, + }; + assert_eq!( + encode_host_line(&observe).unwrap(), + b"{\"type\":\"observe\",\"owner\":{\"incarnation\":\"incarnation\",\"claim\":\"claim\"},\"bindingId\":\"binding\",\"registration\":\"registration\",\"demandWatermark\":7}\n" + ); } #[test] - fn runtime_frames_have_exact_json_shape_and_padded_base64() { + fn periodic_publish_has_exact_flat_json_shape_and_padded_base64() { assert_eq!( encode_runtime_line(&publish(b"one byte")).unwrap(), - b"{\"type\":\"publish\",\"owner\":{\"incarnation\":\"incarnation\",\"claim\":\"claim\"},\"bindingId\":\"binding\",\"registration\":\"registration\",\"schemaId\":\"schema.v1\",\"mediaType\":\"application/json\",\"bytes\":\"b25lIGJ5dGU=\",\"topics\":[\"selected\"],\"observedAt\":\"2026-08-30T00:00:00Z\"}\n" + b"{\"type\":\"publish\",\"owner\":{\"incarnation\":\"incarnation\",\"claim\":\"claim\"},\"bindingId\":\"binding\",\"registration\":\"registration\",\"schemaId\":\"schema.v1\",\"mediaType\":\"application/json\",\"bytes\":\"b25lIGJ5dGU=\",\"topics\":[\"selected\"]}\n" ); let health = RuntimeMessage::Health { owner: owner(), @@ -849,13 +1052,82 @@ mod tests { ); } + #[test] + fn abi_3_publish_decodes_with_or_without_deprecated_observed_at_only() { + let without_observed_at = b"{\"type\":\"publish\",\"owner\":{\"incarnation\":\"incarnation\",\"claim\":\"claim\"},\"bindingId\":\"binding\",\"registration\":\"registration\",\"schemaId\":\"schema.v1\",\"mediaType\":\"application/json\",\"bytes\":\"b25lIGJ5dGU=\",\"topics\":[\"selected\"]}\n"; + let with_observed_at = b"{\"type\":\"publish\",\"owner\":{\"incarnation\":\"incarnation\",\"claim\":\"claim\"},\"bindingId\":\"binding\",\"registration\":\"registration\",\"schemaId\":\"schema.v1\",\"mediaType\":\"application/json\",\"bytes\":\"b25lIGJ5dGU=\",\"topics\":[\"selected\"],\"observedAt\":\"2026-08-30T00:00:00Z\"}\n"; + let unknown_field = b"{\"type\":\"publish\",\"owner\":{\"incarnation\":\"incarnation\",\"claim\":\"claim\"},\"bindingId\":\"binding\",\"registration\":\"registration\",\"schemaId\":\"schema.v1\",\"mediaType\":\"application/json\",\"bytes\":\"b25lIGJ5dGU=\",\"topics\":[\"selected\"],\"extra\":true}\n"; + + assert_eq!( + decode_runtime_line(without_observed_at).unwrap(), + publish(b"one byte") + ); + assert_eq!( + decode_runtime_line(with_observed_at).unwrap(), + publish(b"one byte") + ); + assert!(matches!( + decode_runtime_line(unknown_field), + Err(ProtocolError::Json(_)) + )); + } + + #[test] + fn observation_results_have_one_atomic_tagged_wire_shape() { + let unchanged = RuntimeMessage::ObservationResult { + owner: owner(), + binding_id: BindingId::new("binding").unwrap(), + registration: RegistrationToken::new("registration").unwrap(), + demand_watermark: 7, + result: ObservationResult::Unchanged, + }; + assert_eq!( + encode_runtime_line(&unchanged).unwrap(), + b"{\"type\":\"observationResult\",\"owner\":{\"incarnation\":\"incarnation\",\"claim\":\"claim\"},\"bindingId\":\"binding\",\"registration\":\"registration\",\"demandWatermark\":7,\"result\":{\"status\":\"unchanged\"}}\n" + ); + + let failed = RuntimeMessage::ObservationResult { + owner: owner(), + binding_id: BindingId::new("binding").unwrap(), + registration: RegistrationToken::new("registration").unwrap(), + demand_watermark: 8, + result: ObservationResult::Failed { + diagnostic: Some("provider unavailable".to_owned()), + }, + }; + assert_eq!( + encode_runtime_line(&failed).unwrap(), + b"{\"type\":\"observationResult\",\"owner\":{\"incarnation\":\"incarnation\",\"claim\":\"claim\"},\"bindingId\":\"binding\",\"registration\":\"registration\",\"demandWatermark\":8,\"result\":{\"status\":\"failed\",\"diagnostic\":\"provider unavailable\"}}\n" + ); + + let mut published = publication(b"one byte"); + published.facts = Some(vec![ResourceFact::current("state", "ready").unwrap()]); + let published = RuntimeMessage::ObservationResult { + owner: owner(), + binding_id: BindingId::new("binding").unwrap(), + registration: RegistrationToken::new("registration").unwrap(), + demand_watermark: 9, + result: ObservationResult::Published { + publication: published, + }, + }; + assert_eq!( + encode_runtime_line(&published).unwrap(), + b"{\"type\":\"observationResult\",\"owner\":{\"incarnation\":\"incarnation\",\"claim\":\"claim\"},\"bindingId\":\"binding\",\"registration\":\"registration\",\"demandWatermark\":9,\"result\":{\"status\":\"published\",\"publication\":{\"schemaId\":\"schema.v1\",\"mediaType\":\"application/json\",\"bytes\":\"b25lIGJ5dGU=\",\"topics\":[\"selected\"],\"facts\":[{\"key\":\"state\",\"after\":\"ready\"}]}}}\n" + ); + assert_eq!( + decode_runtime_line(&encode_runtime_line(&published).unwrap()).unwrap(), + published + ); + } + #[test] fn fact_wire_shape_distinguishes_omission_from_explicit_null() { let mut message = publish(b"fact"); - let RuntimeMessage::Publish { facts, .. } = &mut message else { + let RuntimeMessage::Publish { publication, .. } = &mut message else { unreachable!(); }; - *facts = Some(vec![ + publication.facts = Some(vec![ ResourceFact::current("state", "ready").unwrap(), ResourceFact::transition("label", None::, Some("added")).unwrap(), ResourceFact::transition("removed", Some("old"), None::).unwrap(), @@ -874,26 +1146,30 @@ mod tests { } #[test] - fn invalid_fact_shapes_and_bounds_are_rejected() { + fn invalid_fact_shapes_and_bounds_are_rejected_for_shared_publications() { let missing_values = b"{\"type\":\"publish\",\"owner\":{\"incarnation\":\"i\",\"claim\":\"c\"},\"bindingId\":\"b\",\"registration\":\"r\",\"schemaId\":\"s\",\"mediaType\":\"m\",\"bytes\":\"\",\"topics\":[],\"facts\":[{\"key\":\"state\"}]}\n"; assert!(matches!( decode_runtime_line(missing_values), Err(ProtocolError::InvalidFacts(FactError::MissingValue)) )); + let demanded_missing_values = b"{\"type\":\"observationResult\",\"owner\":{\"incarnation\":\"i\",\"claim\":\"c\"},\"bindingId\":\"b\",\"registration\":\"r\",\"demandWatermark\":1,\"result\":{\"status\":\"published\",\"publication\":{\"schemaId\":\"s\",\"mediaType\":\"m\",\"bytes\":\"\",\"topics\":[],\"facts\":[{\"key\":\"state\"}]}}}\n"; + assert!(matches!( + decode_runtime_line(demanded_missing_values), + Err(ProtocolError::InvalidFacts(FactError::MissingValue)) + )); assert!(ResourceFact::current("", "value").is_err()); assert!(ResourceFact::current("state", "two\nlines").is_err()); assert!(ResourceFact::current("x".repeat(MAX_FACT_KEY_BYTES + 1), "value").is_err()); assert!(ResourceFact::current("state", "x".repeat(MAX_FACT_VALUE_BYTES + 1)).is_err()); + let too_many = (0..=MAX_FACTS) + .map(|index| ResourceFact::current(format!("key-{index}"), "value").unwrap()) + .collect(); let mut message = publish(b"facts"); - let RuntimeMessage::Publish { facts, .. } = &mut message else { + let RuntimeMessage::Publish { publication, .. } = &mut message else { unreachable!(); }; - *facts = Some( - (0..=MAX_FACTS) - .map(|index| ResourceFact::current(format!("key-{index}"), "value").unwrap()) - .collect(), - ); + publication.facts = Some(too_many); assert!(matches!( encode_runtime_line(&message), Err(ProtocolError::InvalidFacts(FactError::TooMany { .. })) @@ -901,7 +1177,7 @@ mod tests { } #[test] - fn decoding_is_strict_about_fields_ids_and_digest_encoding() { + fn decoding_is_strict_about_fields_ids_digest_and_obsolete_protocol() { let unknown_message_field = b"{\"type\":\"health\",\"owner\":{\"incarnation\":\"i\",\"claim\":\"c\"},\"state\":\"ready\",\"extra\":true}\n"; assert!(matches!( decode_runtime_line(unknown_message_field), @@ -922,6 +1198,46 @@ mod tests { decode_host_line(uppercase_digest), Err(ProtocolError::Json(_)) )); + let unknown_result_field = b"{\"type\":\"observationResult\",\"owner\":{\"incarnation\":\"i\",\"claim\":\"c\"},\"bindingId\":\"b\",\"registration\":\"r\",\"demandWatermark\":1,\"result\":{\"status\":\"unchanged\",\"extra\":true}}\n"; + assert!(matches!( + decode_runtime_line(unknown_result_field), + Err(ProtocolError::Json(_)) + )); + let unknown_publication_field = b"{\"type\":\"observationResult\",\"owner\":{\"incarnation\":\"i\",\"claim\":\"c\"},\"bindingId\":\"b\",\"registration\":\"r\",\"demandWatermark\":1,\"result\":{\"status\":\"published\",\"publication\":{\"schemaId\":\"s\",\"mediaType\":\"m\",\"bytes\":\"\",\"topics\":[],\"extra\":true}}}\n"; + assert!(matches!( + decode_runtime_line(unknown_publication_field), + Err(ProtocolError::Json(_)) + )); + + let obsolete_settlement = b"{\"type\":\"observationSettled\",\"owner\":{\"incarnation\":\"i\",\"claim\":\"c\"},\"bindingId\":\"b\",\"registration\":\"r\",\"demandWatermark\":1,\"outcome\":\"unchanged\"}\n"; + assert!(matches!( + decode_runtime_line(obsolete_settlement), + Err(ProtocolError::Json(_)) + )); + } + + #[test] + fn observation_watermarks_must_be_positive() { + let zero_observe = b"{\"type\":\"observe\",\"owner\":{\"incarnation\":\"i\",\"claim\":\"c\"},\"bindingId\":\"b\",\"registration\":\"r\",\"demandWatermark\":0}\n"; + assert!(matches!( + decode_host_line(zero_observe), + Err(ProtocolError::InvalidDemandWatermark) + )); + let zero_result = b"{\"type\":\"observationResult\",\"owner\":{\"incarnation\":\"i\",\"claim\":\"c\"},\"bindingId\":\"b\",\"registration\":\"r\",\"demandWatermark\":0,\"result\":{\"status\":\"unchanged\"}}\n"; + assert!(matches!( + decode_runtime_line(zero_result), + Err(ProtocolError::InvalidDemandWatermark) + )); + let observe = HostMessage::Observe { + owner: owner(), + binding_id: BindingId::new("binding").unwrap(), + registration: RegistrationToken::new("registration").unwrap(), + demand_watermark: 0, + }; + assert!(matches!( + encode_host_line(&observe), + Err(ProtocolError::InvalidDemandWatermark) + )); } #[test] @@ -1022,6 +1338,26 @@ mod tests { decode_runtime_line(&health_line), Err(ProtocolError::HealthDetailTooLarge { .. }) )); + + let oversized_diagnostic = RuntimeMessage::ObservationResult { + owner: owner(), + binding_id: BindingId::new("binding").unwrap(), + registration: RegistrationToken::new("registration").unwrap(), + demand_watermark: 1, + result: ObservationResult::Failed { + diagnostic: Some("x".repeat(MAX_OBSERVATION_DIAGNOSTIC_BYTES + 1)), + }, + }; + assert!(matches!( + encode_runtime_line(&oversized_diagnostic), + Err(ProtocolError::ObservationDiagnosticTooLarge { .. }) + )); + let mut diagnostic_line = serde_json::to_vec(&oversized_diagnostic).unwrap(); + diagnostic_line.push(b'\n'); + assert!(matches!( + decode_runtime_line(&diagnostic_line), + Err(ProtocolError::ObservationDiagnosticTooLarge { .. }) + )); assert!(RuntimeIncarnation::new("x".repeat(MAX_OPAQUE_ID_BYTES + 1)).is_err()); } @@ -1039,10 +1375,10 @@ mod tests { Err(ProtocolError::InvalidHealthScope) )); let mut duplicate_topics = publish(b"bytes"); - let RuntimeMessage::Publish { topics, .. } = &mut duplicate_topics else { + let RuntimeMessage::Publish { publication, .. } = &mut duplicate_topics else { unreachable!(); }; - *topics = vec!["same".to_owned(), "same".to_owned()]; + publication.topics = vec!["same".to_owned(), "same".to_owned()]; assert!(matches!( encode_runtime_line(&duplicate_topics), Err(ProtocolError::InvalidTopics(_)) diff --git a/docs/vrs/.decisions/0014-resource-profiles-are-state-first-read-and-observe-capabilities.md b/docs/vrs/.decisions/0014-resource-profiles-are-state-first-read-and-observe-capabilities.md index e15049e9..cd3a7e3a 100644 --- a/docs/vrs/.decisions/0014-resource-profiles-are-state-first-read-and-observe-capabilities.md +++ b/docs/vrs/.decisions/0014-resource-profiles-are-state-first-read-and-observe-capabilities.md @@ -2,76 +2,135 @@ Status: accepted -Johannes selected this direction through ten recorded interview rounds on 2026-08-29 and confirmed the combined design before VRS work began. Event-first authority remains a separate exploration in [issue #376](https://github.com/compoundingtech/st2/issues/376). +Johannes selected this direction through ten recorded interview rounds on +2026-08-29 and confirmed the combined design before VRS work began. Event-first +authority remains a separate exploration in [issue #376](https://github.com/compoundingtech/st2/issues/376). ## Context -Resource Profiles previously mapped an opaque URI to one contained local carrier. Filesystem resync could notify a live agent after local bytes changed, but the profile had no way to observe a remote Resource, publish current state, validate profile-specific attention settings, report runtime health, or catch an agent up after delivery was unavailable. - -GitHub pull requests and issues made the gap concrete. Mergeability, CI, reviews, comments, and lifecycle state change independently. Raw webhook or check events can be duplicated, reordered, missed, or noisy. GitHub does not automatically redeliver failed webhook deliveries. Existing st2 stream ingress intentionally has bounded deduplication, and filesystem resync intentionally has no catch-up. Treating either delivery path as a complete canonical event log would claim guarantees that neither substrate provides. - -Two prototypes bounded the design. A synthetic 16-observation pull-request sequence showed that equal-state suppression removed only two candidate wakes, while provider-aware semantic filtering reduced the sequence to seven actionable or four blocking/terminal wakes. A Rust state-space model then checked 97,656 lifecycle sequences and found that storing a pending historical digest can point a resumed agent at stale state; one pending-relevance bit plus the current snapshot digest is sufficient. +A Resource Profile maps an opaque Resource URI to a contained local carrier. +That passive mapping alone cannot observe a remotely changing Resource, +publish its current state, validate profile-specific attention settings, report +runtime health, or catch an agent up after delivery was unavailable. + +GitHub pull requests and issues make the gap concrete. Mergeability, CI, +reviews, comments, and lifecycle state change independently. Raw webhook or +check events can be duplicated, reordered, missed, or noisy, and failed webhook +deliveries are not necessarily replayed. Existing st2 delivery paths likewise +do not promise a complete canonical event log. Treating any of those inputs as +authoritative would claim ordering and replay guarantees the substrates do not +provide. + +Two prototypes bounded the design. A synthetic 16-observation pull-request +sequence showed that equal-state suppression removed only two candidate wakes, +while provider-aware semantic filtering reduced the sequence to seven +actionable or four blocking/terminal wakes. A Rust state-space model then +checked 97,656 lifecycle sequences and found that storing a pending historical +digest can point a resumed agent at stale state; one pending-relevance +condition plus the current snapshot digest is sufficient. ## Decision -1. One atomic current snapshot per Resource binding is canonical. Provider events, polls, and webhooks are observations used to reconcile that snapshot. Notifications are invalidations, not a complete event log. -2. st2 owns a provider-neutral read-and-observe contract: descriptor execution, selector validation, runtime lifecycle, host-owned snapshot publication, health, bounded delivery, and current-state catch-up. Downstream profiles own URI semantics, provider authentication, provider observation, reconciliation, snapshot schema, semantic topics, and defaults. -3. The bounded profile module exports a versioned descriptor containing capabilities, selector schema, default selector, published topics, runtime topology, snapshot media type, and snapshot schema identity. A binding may provide validated selector configuration; omission uses the profile default. -4. The first contract exposes one atomic snapshot rather than named facets or a generation manifest. A changed snapshot emits a thin invalidation containing binding identity, current digest, and selected semantic topics. It carries no snapshot bytes or rendered summary. -5. The profile implementation chooses the most efficient provider-native observation mechanism. It may use push, polling, or a hybrid and retains any provider cursor or repair state. The closed resolver module gains no ambient network or credential imports. -6. The descriptor declares shared or per-binding runtime topology. Both use one normalized host protocol and the same per-binding delivery state. -7. When delivery is unavailable, st2 retains only `pending_relevant_change` beside current and last-delivered digests. Resume emits at most one invalidation for current state. -8. The initial capability set stops at read and observe. Provider mutations, actions, approvals, and a canonical event log require separate research and design. +1. One atomic current snapshot per Resource binding is canonical. Provider + events, polls, webhooks, and demands are observations used to reconcile that + snapshot. Notifications are invalidations, not a complete event log. +2. st2 owns a provider-neutral read-and-observe contract: descriptor execution, + selector validation, runtime lifecycle, host-owned publication, health, + bounded delivery, and current-state catch-up. Downstream profiles own URI + semantics, provider authentication, observation, reconciliation, snapshot + schema, semantic topics and facts, and selector defaults. +3. Descriptor ABI 3 contains the capabilities, selector schema, default + selector, published topics, runtime topology, snapshot media type, and + snapshot schema identity. A binding may provide validated selector + configuration; omission uses the profile default. +4. `Publication` is the one reusable publication value. It contains schema + identity, media type, snapshot bytes, topics, and optional bounded typed + facts. Periodic `Publish` and the demanded `Published` result carry that same + value through one host acceptance, digest, publication, relevance, and + catch-up path. +5. The profile implementation chooses its provider-native observation + mechanism. It may use push, polling, or a hybrid and retains its provider + cursor, conditional cache, rate limits, backoff, and repair state. The closed + resolver module gains no ambient network or credential imports. +6. The descriptor declares shared or per-binding runtime topology. Both use one + normalized host protocol, directional owner fencing, registration fencing, + and the same per-binding publication and delivery state. +7. Demand observation is an explicit, deny-by-default runtime capability. For + an enabled active registration, the host may send `Observe` with a positive + demand watermark. The runtime answers that demand with exactly one atomic + `ObservationResult` for the same owner, binding, registration, and watermark: + `Unchanged`, `Failed` with an optional bounded diagnostic, or `Published` + with a `Publication`. There is no separate publication/settlement pair and + no host timestamp in the protocol. +8. Demand is a level-triggered scheduling hint, not a provider-specific + reconcile command or provider write. One in-flight dispatch and one latest + trailing watermark coalesce bursts without dropping demand that arrives + during observation. Exact result, registration replacement, or + provider-process failure evidence closes accepted work; clocks do not. +9. A client wait bound limits only that client's wait. Disconnect or expiry + does not cancel accepted demand, retract the supervisor's obligation, or + participate in provider scheduling. +10. When delivery is unavailable, st2 retains the current and last-delivered + digests, one pending-relevance condition, and the latest relevant selected + topics and facts. Resume emits at most one invalidation for then-current + state with that semantic envelope. +11. Read and observe do not authorize provider mutations. Actions, approvals, + and a canonical event log require separate authority, idempotency, audit, + and result-delivery designs. ## Options | Option | Why it was not selected | | --- | --- | -| Canonical Resource event log | Most principled long-term possibility, but current delivery substrates do not provide complete replay, ordering, retention, cursor recovery, or gap repair. Tracked separately in issue #376. | +| Canonical Resource event log | Current delivery substrates do not provide complete replay, ordering, retention, cursor recovery, or gap repair. Tracked separately in issue #376. | | Hybrid snapshot plus several semantic streams | Provides transition fidelity and independent policies, but adds public stream contracts before notification-volume evidence proves that need. | | Keep st2 resolver-only | Existing primitives can be composed downstream, but every profile would reimplement lifecycle, validation, health, catch-up, and publication. | -| One event or snapshot facet per semantic topic | Enables selective reads but introduces cross-facet consistency and generation lifecycle. One snapshot is sufficient for the first evidence-backed use case. | -| Deliver profile-rendered summaries | May save a read, but increases prompt noise and creates redaction/rendering obligations without measured token savings. | +| One event or snapshot facet per semantic topic | Enables selective reads but introduces cross-facet consistency and generation lifecycle. One atomic snapshot is sufficient for the evidence-backed use case. | +| Deliver snapshot bytes in invalidations | Duplicates the canonical carrier, increases delivery size, and weakens the state-first read boundary. Bounded topics and typed facts convey why a current-state read matters. | | Preserve every update while delivery is unavailable | Produces a backlog that conflicts with state-first authority and the explicit noise constraint. | -| Standardize provider actions now | No concrete action workflow, authority model, approval contract, or idempotency prototype grounds a generic action API. | +| Separate `Publish` and demand settlement frames | Allows settlement and publication to disagree or be lost independently. One tagged atomic demand result has one acceptance point and one outcome. | +| Host-selected observation mechanics | Conflates generic demand with provider-specific reconciliation and transfers cursor, cache, rate-limit, and backoff policy to st2. | +| Standardize provider actions | No generic action workflow, authority model, approval contract, or idempotency evidence grounds such an API. | ## Evidence and Argument - [GitHub attention-filter prototype](../07-resource-profile/.experiments/2026-08-29-github-attention-filter-prototype.md) - [Smart Resource lifecycle state-space prototype](../07-resource-profile/.experiments/2026-08-29-smart-resource-lifecycle-prototype.md) +- [Selector and runtime protocol prototype](../07-resource-profile/.experiments/2026-08-29-selector-and-runtime-protocol-prototype.md) - [Existing stream differentiation experiment](../04-stream/.experiments/2026-08-20-pipes-event-model-differentiation.md) - [Existing Resource Profile boundary comparison](../07-resource-profile/.experiments/2026-08-26-plugin-boundary-comparison.md) - [GitHub webhook best practices](https://docs.github.com/en/webhooks/using-webhooks/best-practices-for-using-webhooks) - [GitHub webhook redelivery](https://docs.github.com/en/webhooks/testing-and-troubleshooting-webhooks/redelivering-webhooks) -State-first authority makes delivery loss and coalescing safe: reconciliation repairs current state, and an invalidation only prompts a read. Provider-aware classification is necessary because byte equality cannot distinguish actionable CI failure from routine progress. Binding selectors preserve consumer control, while descriptor defaults avoid declaration noise. Host-owned publication prevents a network-capable runtime from bypassing containment or replacing the carrier through an unsafe filesystem path. +State-first authority makes delivery loss and coalescing safe: reconciliation +repairs current state, and an invalidation only prompts a read. Provider-aware +classification is necessary because byte equality cannot distinguish +actionable failure from routine progress. Binding selectors preserve consumer +control, while descriptor defaults avoid declaration noise. Host-owned +publication prevents a network-capable runtime from bypassing containment. + +Pending notification is a level condition, not an event identity. If a relevant +change occurs while delivery is unavailable, a later publication may advance +the current digest. Catch-up therefore combines pending relevance and its +bounded semantic envelope with the authoritative current digest rather than +retaining a historical digest or transition backlog. -The lifecycle prototype supplies the main complexity reduction. A pending notification is a level condition, not an event identity. If a relevant change occurred while delivery was unavailable, any later snapshot update advances the current digest. Resume therefore points at current state by combining one pending bit with the authoritative digest. Shared and per-binding runtime topology do not need separate delivery reducers. +The same reasoning governs demand. A watermark identifies host work, not +provider history. An atomic tagged result cannot race a separate publication +against settlement, and owner plus registration fencing prevents a replaced +runtime or binding generation from satisfying current work. ## Consequences -- Resource Profile ABI evolution becomes immediate rather than hypothetical. Descriptor and host-protocol compatibility require conformance fixtures before third-party modules or runtimes are supported. -- Agent Spec needs one profile-specific selector encoding. Its exact KDL-to-schema representation remains open and blocks implementation of binding overrides, not profile defaults. -- st2 gains a trusted host-process boundary for observable runtimes. The catalog operator, not the wasm guest or Resource URI, selects the executable and its deployed credentials and egress. -- The existing built-in `resync` stream can carry smart Resource invalidations keyed by binding; no new stream family or delivery plane is required. -- Profile implementations may optimize aggressively for native real-time observation, but st2 judges only convergent publication and explicit health. It does not prescribe polling intervals or webhook repair. -- Actions and event-first authority remain out of scope rather than being smuggled into descriptor extensibility. - -## Amendment 1: selector encoding and runtime fencing - -On 2026-08-29 Johannes selected raw JSON in a KDL `selector` property (Q12) -after a runnable comparison of concise, nested, and adversarial selectors. -Normalized JSON remains the cross-format and runtime value. KDL is only an -authoring representation; its canonical renderer chooses the smallest safe raw -string hash fence. This resolves the open selector-encoding consequence above. - -The same experiment resolved runtime restart ownership without adding another -lifecycle protocol. Each process incarnation receives a directional owner -claim, each binding registration receives a token, and host acceptance requires -both to match. A new claim fences all prior output and clears registrations. -Shared and per-binding topologies use the same reducer. - -The normalized wire protocol contains only `register`, `unregister`, `publish`, -and `health`. EOF and existing supervisor lifecycle replace `shutdown`; -implementation-owned observation replaces host `reconcile`. Evidence: -[selector and runtime protocol prototype](../07-resource-profile/.experiments/2026-08-29-selector-and-runtime-protocol-prototype.md). +- Descriptor ABI and host-protocol compatibility require conformance fixtures + before independently released third-party modules or runtimes are supported. +- st2 gains a trusted host-process boundary for observable runtimes. The + catalog operator, not the wasm guest or Resource URI, selects the executable + and its deployed credentials, capabilities, and egress. +- The built-in `resync` stream carries Resource invalidations keyed by binding; + no new stream family or delivery plane is required. +- Profile implementations may optimize aggressively for native real-time + observation, but st2 judges convergent publication, exact demand results, and + explicit health. It does not prescribe polling intervals or webhook repair. +- Actions and event-first authority remain out of scope rather than being + smuggled into descriptor or runtime extensibility. diff --git a/docs/vrs/06-observability/spec.md b/docs/vrs/06-observability/spec.md index 29e4d8c2..b6305efd 100644 --- a/docs/vrs/06-observability/spec.md +++ b/docs/vrs/06-observability/spec.md @@ -173,12 +173,19 @@ the instruments; every record call early-outs unless a meter provider is install | `message_deliveries_total` | counter | `result` = `pass` \| `fail` | | `crash_loops_total` | counter | — | | `driver_diagnostic_transitions_total` | counter | `stage`, `reason`, `source`, `support`, `outcome = failure | recovery` (all closed enums) | +| `resource_observe_requests_total` | counter | `outcome` = `accepted` \| `backpressured` \| `settledUnchanged` \| `settledChanged` \| `settledFailed` \| `absentBinding` \| `staleGeneration` \| `providerUnavailable` \| `other` | +| `resource_observe_dispatch_seconds` | histogram | — | +| `resource_observe_settle_seconds` | histogram | — | | `reconcile_pass_duration_seconds` | histogram | — | | `session_start_duration_seconds` | histogram | — | -The duration histograms use seconds-scale explicit buckets (`1ms … 10s`, see -`DURATION_BUCKET_BOUNDARIES` in `src/telemetry.rs`) instead of the SDK's millisecond-tuned -defaults, so sub-second passes and spawns stay distinguishable. +All duration histograms share seconds-scale explicit bucket boundaries +`0.001`, `0.005`, `0.01`, `0.025`, `0.05`, `0.1`, `0.25`, `0.5`, `1`, +`2.5`, `5`, and `10` (`DURATION_BUCKET_BOUNDARIES` in `src/telemetry.rs`) +instead of the SDK's millisecond-tuned defaults. This keeps sub-second +reconcile passes, spawns, observe dispatches, and settlements distinguishable. +Observe metric statuses use the same camelCase durable-wire spelling; kebab-case +is reserved for human CLI text. Scope notes: passes are counted at all three `st2.reconcile_pass` sites (catalog loop pass, one-shot up, and the single-file spec path — `reconcile_pass_specs_with_sessions`, which now diff --git a/docs/vrs/07-resource-profile/open-questions.md b/docs/vrs/07-resource-profile/open-questions.md index c6a57286..db43114a 100644 --- a/docs/vrs/07-resource-profile/open-questions.md +++ b/docs/vrs/07-resource-profile/open-questions.md @@ -1,23 +1,25 @@ # Resource Profile open questions The resolver registry, wasm-only pure module boundary, transactional ownership -of catalog-relative modules, and state-first read-and-observe direction are -accepted and therefore are not open questions. +of catalog-relative modules, state-first publication authority, typed semantic +facts, and atomic demand-observation result are accepted and therefore are not +open questions. Demand is explicitly capability-gated; fenced by owner, +registration, and watermark; coalesced to one in-flight plus one trailing +dispatch; and not cancelled by client wait expiry. -DQ-P3 is resolved by the raw JSON `selector` property selected in Q12 and -proved by the selector round-trip prototype. DQ-P4 is resolved by treating -first readable state as a relevant state transition when the publication names -a selected topic. DQ-P5 is resolved by explicit 16 KiB selector, 2 MiB protocol -line, 1 MiB snapshot, and 16 KiB health-detail bounds; representative st2 issue -and pull payloads remained below 41 KiB per item, leaving substantial space for -normalized reviews and check state without permitting unbounded allocation. -DQ-P6 is resolved by one directional owner claim per runtime incarnation, one -token per binding registration, EOF-owned termination, and the shared ownership -reducer proved by the restart state-space prototype. Evidence for selector and -runtime ownership is in the +DQ-P3 is resolved by the raw JSON `selector` property and its round-trip +prototype. DQ-P4 is resolved by treating initial readable state as a relevant +state transition when the publication names a selected topic. DQ-P5 is resolved +by explicit 16 KiB selector, 2 MiB protocol-line, 1 MiB snapshot, 16 KiB health +detail and failed-result diagnostic, and typed-fact bounds. Representative st2 +issue and pull payloads remained below 41 KiB per item, leaving substantial +space for normalized reviews and check state without permitting unbounded +allocation. DQ-P6 is resolved by one directional owner claim per runtime +incarnation, one token per binding registration, EOF-owned termination, and the +shared ownership reducer. Evidence for selector and runtime ownership is in the [selector and runtime protocol experiment](./.experiments/2026-08-29-selector-and-runtime-protocol-prototype.md). -- **DQ-P1 ABI compatibility.** Descriptor ABI v2 makes version selection +- **DQ-P1 ABI compatibility.** Descriptor ABI 3 makes version selection explicit, but old-module/new-host, new-module/old-host, and runtime-protocol compatibility are not yet proven. Resolve before independently released third-party modules or runtimes with a compatibility matrix, frozen fixtures, diff --git a/docs/vrs/07-resource-profile/requirements.md b/docs/vrs/07-resource-profile/requirements.md index e5dfbbe9..73497381 100644 --- a/docs/vrs/07-resource-profile/requirements.md +++ b/docs/vrs/07-resource-profile/requirements.md @@ -9,18 +9,11 @@ that st2 can observe. It refines [`06-resync`](../06-resync/requirements.md) without moving scheme ownership into st2 or making successful resolution a condition of agent launch. -Johannes selected the registry/SDK shape (decision Q8), a wasm-only resolver -foundation after the measured three-way comparison (decision Q10), and -transactional ownership of catalog-relative modules (decision Q14). The -accepted rationale is recorded in +The accepted resolver registry, wasm boundary, and transactional ownership are +recorded in [decision 0009](../.decisions/0009-resource-profiles-use-a-feature-gated-wasm-boundary.md). - -On 2026-08-29 Johannes extended the subsystem from passive local resolution to -a generic read-and-observe lifecycle for remotely changing Resources. Ten -recorded interview rounds selected state-first authority, profile-defined -schemas and defaults with binding selectors, one atomic snapshot, thin -invalidations, implementation-selected observation and runtime topology, and -one latest-state catch-up. The direction is recorded in +The state-first read-and-observe authority, atomic publication and demand +result, typed semantic envelope, and latest-state catch-up are recorded in [decision 0014](../.decisions/0014-resource-profiles-are-state-first-read-and-observe-capabilities.md). ## Assumptions @@ -39,11 +32,12 @@ one latest-state catch-up. The direction is recorded in complete event log, and no consumer may require every provider transition. - **PROFILE-A05 Downstream observation semantics:** A profile implementation owns provider authentication, observation, reconciliation, semantic topics, - snapshot schema, and selector defaults. st2 owns only the generic lifecycle, - validation, publication, delivery, health, and containment contracts. -- **PROFILE-A06 Read-and-observe scope:** The first capability contract does - not mutate provider state or standardize actions. Comments, CI reruns, label - changes, close, merge, approval, and other provider writes require a separate + typed-fact meaning, snapshot schema, and selector defaults. st2 owns the + generic lifecycle, validation, atomic publication and demand result, + coalescing, fencing, delivery, health, and containment contracts. +- **PROFILE-A06 Read-and-observe scope:** Read and observe do not mutate + provider state or standardize actions. Comments, CI reruns, label changes, + close, merge, approval, and other provider writes require a separate authority, approval, idempotency, audit, and result-delivery design. ## Acceptable Tradeoffs @@ -52,9 +46,9 @@ one latest-state catch-up. The direction is recorded in and compile-time cost is accepted in builds that enable `wasm-resolver` so the sandbox complexity is absorbed once. Default builds retain the baseline dependency and binary surface. -- **PROFILE-T02 Owned guest ABI:** st2 owns a small core-wasm ABI and its future - compatibility burden. Avoiding WASI and the component model keeps the initial - capability surface closed, but ABI evolution must be explicit. +- **PROFILE-T02 Owned guest ABI:** st2 owns core-wasm descriptor ABI 3 and its + compatibility burden. Avoiding WASI and the component model keeps the + capability surface closed, but ABI evolution must remain explicit. - **PROFILE-T03 Stateless calls:** Successful compilations and unchanged compilation failures share a bounded cache, while each successful resolution receives a fresh store and instance. The extra instantiation cost is accepted @@ -69,8 +63,8 @@ one latest-state catch-up. The direction is recorded in webhooks, or a hybrid. The profile implementation may use the most efficient provider-native mechanism, accepting responsibility for convergence, backpressure, rate limits, and any provider cursor or repair state. -- **PROFILE-T06 One snapshot rather than facets:** The first contract rewrites - one atomic profile-defined snapshot even when provider facets change +- **PROFILE-T06 One snapshot rather than facets:** The contract rewrites one + atomic profile-defined snapshot even when provider facets change independently. This avoids generation manifests, facet consistency, and retention machinery until measured payload or read costs justify them. - **PROFILE-T07 Schema execution:** Discovering profile capabilities, selector @@ -163,12 +157,12 @@ one latest-state catch-up. The direction is recorded in ### Must describe and validate observable capabilities - **PROFILE-R12 Versioned profile descriptor:** A profile module exposes one - bounded, versioned descriptor in addition to resolution. The descriptor - declares supported capabilities, selector schema, semantic topic vocabulary, - default selector value, runtime topology, snapshot media type and schema - identity, and ABI version. The host validates the descriptor under the same - fuel, memory, output, import, and failure isolation as resolution. Unknown - required capabilities or ABI versions fail that profile locally. + bounded descriptor in addition to resolution. Descriptor ABI 3 declares + supported capabilities, selector schema, semantic topic vocabulary, default + selector value, runtime topology, snapshot media type, and snapshot schema + identity. The host validates the descriptor under the same fuel, memory, + output, import, and failure isolation as resolution. Unknown required + capabilities or ABI versions fail that profile locally. - **PROFILE-R13 Validated binding selectors:** An observable Resource binding may carry profile-specific selector configuration. Absence means the descriptor's default. KDL encodes the value as compact JSON in a `selector` @@ -182,51 +176,100 @@ one latest-state catch-up. The direction is recorded in ### Must publish one canonical current snapshot - **PROFILE-R14 Atomic snapshot authority:** Each active observable binding has - at most one profile-defined canonical current snapshot. Publication replaces - the snapshot atomically, records its content digest and schema identity, and - never exposes partial bytes. Equal-byte publication is a no-op. A first - successful publication with at least one selected topic schedules the same - superseding invalidation as a later relevant change. The snapshot remains the - authority after missed, duplicated, reordered, or coalesced provider - observations. + at most one profile-defined canonical current snapshot. `Publication` is the + reusable payload for every publication form and contains schema identity, + media type, snapshot bytes, semantic topics, and optional ordered typed + facts. The host validates one complete `Publication`, computes its content + digest from accepted bytes, replaces the snapshot atomically, and never + exposes partial bytes. Periodic `Publish` and demand-result `Published` + traverse the same acceptance, digest, relevance, and catch-up core. Equal + bytes do not create a state transition. The first accepted publication with + at least one selected topic schedules the same superseding invalidation as a + later relevant change. The snapshot remains authoritative after missed, + duplicated, reordered, or coalesced provider observations. - **PROFILE-R15 Implementation-owned observation:** A profile implementation - chooses polling, push, native subscription, or a hybrid and may retain its own - provider cursor. st2 standardizes registration, atomic snapshot publication, - backpressure, cancellation, and health outcomes but does not send - observation-specific reconcile commands or prescribe the provider mechanism. - Provider payloads never bypass snapshot publication to become canonical - delivery records. -- **PROFILE-R16 Declared runtime topology:** The descriptor declares either one - shared runtime per catalog and exact scheme or one runtime per active binding. - Both modes use one host protocol and per-binding lifecycle state. Each runtime - incarnation receives a directional owner claim; each binding registration - receives a token. The host rejects output unless both still match current - state. EOF and existing supervisor process lifecycle own termination and - restart. Shared-runtime failure may affect observation for many bindings but - must report health per binding; per-binding failure remains local. + chooses polling, push, native subscription, or a hybrid and retains its own + provider mechanism, cursor, conditional cache, rate-limit state, backoff, and + repair policy. A generic demand may pull an eligible observation forward but + never selects the provider mechanism, resets provider state, or becomes a + provider-specific reconcile command. Provider payloads never bypass + `Publication` to become canonical delivery records, and demand observation + never authorizes a provider write. +- **PROFILE-R16 Declared runtime topology and fencing:** The descriptor declares + either one shared runtime per catalog and exact scheme or one runtime per + active binding. Both modes use one host protocol and per-binding lifecycle + state. Each runtime incarnation receives a directional owner claim; each + binding registration receives a token. The host accepts or addresses output + only while owner, binding, and registration match current state. EOF and the + supervisor process lifecycle own termination and restart. Shared-runtime + failure may affect observation for many bindings but reports health per + binding; per-binding failure remains local. - **PROFILE-R16A Finite protocol and publication bounds:** A selector's canonical compact JSON is at most 16 KiB. One encoded runtime-protocol line is at most 2 MiB including its newline. Decoded snapshot bytes are at most 1 MiB. - Health detail is at most 16 KiB of UTF-8. One publication carries at most 32 - ordered facts; each fact key is at most 128 bytes and each before/after value - is at most 1 KiB of printable single-line UTF-8. st2 rejects an oversized - value without truncation and contains the failure to the affected runtime or - binding. + Health detail and a failed demand diagnostic are each at most 16 KiB of + UTF-8. One `Publication` carries at most 32 ordered facts; each fact key is at + most 128 bytes and each before/after value is at most 1 KiB of printable + single-line UTF-8. st2 rejects an oversized value without truncation and + contains the failure to the affected runtime or binding. +- **PROFILE-R16B Declared atomic demand:** Demand observation is explicitly + declared and denied by default. Only a runtime declaration with the `demand` + capability may receive `Observe`. Each `Observe` carries a positive demand + watermark and current owner, binding, and registration fences. The runtime + answers exactly once for that demand with one correspondingly fenced + `ObservationResult`: `Unchanged`; `Failed` with an optional bounded + diagnostic; or `Published` with one complete `Publication`. There is no + separate demand publication and settlement, digest supplied by the runtime, + or protocol observation timestamp. +- **PROFILE-R16C Coalesced, non-cancelling demand:** For one active + registration, st2 keeps at most one demand dispatch in flight and one latest + trailing watermark. Demand accepted during an in-flight observation survives + its result and coalesces into the trailing dispatch. Only an exact atomic + result, replacement of its fenced registration, or provider-process failure + closes accepted work; no clock participates in correctness. `Published` + settles as `settledChanged` with the host-computed accepted-publication + digest, including when equal bytes create no state transition or resync + delivery emission fails after the snapshot and catch-up transaction commits. + A missing active binding maps to `absentBinding`; a binding whose runtime did + not declare demand also maps to `absentBinding` with the explicit diagnostic + `the profile runtime does not declare the demand capability`. A client + generation older than the resident supervisor maps to `staleGeneration`; a + newer generation remains queued until supervisor refresh. Provider failure + maps to `providerUnavailable`. Client disconnect or wait expiry does not + cancel accepted work, retract it, or alter the runtime's observation schedule. +- **PROFILE-R16D Durable demand intent:** Observe request and receipt records + carry the exact schema identities `st2.resource-observe-request.v1` and + `st2.resource-observe-receipt.v1`. They are private to one supervisor scope + and bounded to 64 KiB each. Durable admission permits at most 256 unresolved + requests per scope; an attempt beyond that cap receives submission + backpressure before it is admitted, and request scanning remains bounded by + the same cap. An admitted request record remains the durable, retryable + intent until a terminal receipt is durably committed; only then may the + request be removed. In-memory enqueue and nonterminal receipts do not + transfer that ownership. + A failed terminal receipt commit retains retryable state and + leaves the request eligible for restart. + Receipt status values use camelCase: `accepted` and `backpressured` are + nonterminal; + `settledUnchanged`, `settledChanged`, `settledFailed`, `absentBinding`, + `staleGeneration`, and `providerUnavailable` are terminal. Only + `settledChanged` carries the host-computed digest of accepted publication + bytes. Provider diagnostics normalize to an optional bounded receipt value. ### Must bound attention and catch up to current state - **PROFILE-R17 Semantic invalidation:** Every Resource invalidation carries the same bounded ordered fact envelope in its durable body and renders at most - three whole facts into a subject of at most 96 Unicode scalars. Observable - profiles may publish facts and semantic topics beside changed snapshot bytes; - st2 validates the facts, applies the binding selector to topics, and retains - both through catch-up. Passive carrier changes publish one `content` topic - and a short digest transition fact. Agent Spec declaration changes publish - ordered binding-label facts for added, removed, and semantically changed - Resource declarations without exposing URIs or reasons; unavailable - declaration parsing falls back to a digest transition fact rather than - dropping the invalidation. Snapshot bytes and provider payloads remain out of - the event. + three whole facts into a subject of at most 96 Unicode scalars. Both periodic + and demand publications may supply facts and semantic topics in + `Publication`; st2 validates the facts, applies the binding selector to + topics, and retains the selected topics and facts through catch-up. Passive + carrier changes publish one `content` topic and a short digest-transition + fact. Agent Spec declaration changes publish ordered binding-label facts for + added, removed, and semantically changed Resource declarations without + exposing URIs or reasons; unavailable declaration parsing falls back to a + digest-transition fact rather than dropping the invalidation. Snapshot bytes + and provider payloads remain in the authoritative carrier, not the event. - **PROFILE-R18 Built-in superseding delivery:** Smart Resource invalidations reuse one built-in per-agent delivery stream and the existing inbox, DING, deduplication, and producer-side supersession machinery. The binding name is @@ -234,10 +277,11 @@ one latest-state catch-up. The direction is recorded in delivery plane. - **PROFILE-R19 Level-triggered catch-up:** Snapshot reconciliation continues while delivery is unavailable. Per binding, st2 retains the current snapshot - digest, the last-delivered digest, and one pending-relevance bit, not a - transition backlog or pending historical digest. When delivery becomes - available, a pending relevant change emits at most one invalidation for the - then-current snapshot digest. + digest, the last-delivered digest, one pending-relevance condition, and the + latest relevant selected topics and facts, not a transition backlog or + pending historical digest. When delivery becomes available, pending relevant + state emits at most one invalidation for the then-current snapshot digest + with that retained semantic envelope. - **PROFILE-R20 Observable health:** st2 reports descriptor, selector, observation, reconciliation, publication, and delivery health separately. Failure degrades only the affected profile runtime or binding, preserves the diff --git a/docs/vrs/07-resource-profile/spec.md b/docs/vrs/07-resource-profile/spec.md index fcb367a0..d163bc73 100644 --- a/docs/vrs/07-resource-profile/spec.md +++ b/docs/vrs/07-resource-profile/spec.md @@ -1,20 +1,19 @@ # Resource Profile spec -This document specifies the Resource Profile registry, resolver SDK boundary, and -wasm execution contract. It builds on -[`requirements.md`](./requirements.md). +This document specifies the Resource Profile registry, resolver SDK boundary, +wasm execution contract, observable runtime protocol, and state-first +publication authority. It builds on [`requirements.md`](./requirements.md). -**Status:** Active - -## Scope +## Ownership and flow This subsystem owns scheme-to-profile registration, the guest ABI, descriptor and selector validation, sandbox budgets, host path containment, observable -runtime lifecycle, atomic snapshot publication, and the handoff of passive and -observable carriers to [`06-resync`](../06-resync/spec.md). It does not own -Resource URI semantics, provider authentication, provider observation strategy, -provider mutation, task launch, or a canonical provider event log. Those remain -downstream profile concerns or explicit non-goals. +runtime lifecycle, atomic periodic publication and demand results, and the +handoff of passive and observable carriers to +[`06-resync`](../06-resync/spec.md). It does not own Resource URI semantics, +provider authentication, provider observation strategy, provider mutation, +task launch, or a canonical provider event log. Those remain downstream +profile concerns or explicit non-goals. ## Architecture (PROFILE-R01..R20) @@ -50,9 +49,9 @@ closed wasm describe() -> capabilities + selector schema/default + topology catalog-trusted host runtime argv ----+ | v provider-native observation -publish(binding-id, bytes, topics, facts) +periodic Publish(Publication) or demanded ObservationResult | - v host validation + contained atomic replacement + v one host validation + digest + atomic publication authority canonical snapshot + current digest | v selector + pending-relevance reducer retaining topics + facts @@ -381,8 +380,8 @@ fresh-instance policy, fuel budget, and no-import rule as `resolve`: ``` `abiVersion` governs the complete descriptor and host protocol. Capabilities -are closed strings known by that ABI version; v2 accepts `resolve`, `read`, and -`observe`. `topics[].name` values are unique, non-empty profile-owned semantic +are closed strings known by that ABI version; ABI 3 accepts `resolve`, `read`, +and `observe`. `topics[].name` values are unique, non-empty profile-owned identifiers. `defaultSelector` must validate against `selectorSchema` and name only published topics. A binding selector is validated against the same schema and topic set before registration. Selector configuration is observation @@ -403,7 +402,7 @@ payload. JSON and TOML Agent Spec forms carry the selector as a native JSON value. All forms lower to the same `serde_json::Value`; KDL spelling is not preserved and cannot change selector semantics. -## Observable runtime declaration and protocol (PROFILE-R15..R16) +## Observable runtime declaration and protocol (PROFILE-R15..R16D) The closed wasm module never receives network, credential, filesystem, process, or clock imports. A profile with `observe` therefore also has one @@ -415,16 +414,20 @@ profile "github-pr" { class "coalesced" runtime { argv "github-resource-runtime" "pr" + capability "demand" } } ``` `runtime` is forbidden unless the descriptor declares `observe`, and `observe` is unusable without `runtime`. The block accepts exactly one -non-empty `argv` child and never invokes a shell. The executable is an external -operator-trusted input; the guest cannot choose or rewrite it. Environment, -credentials, egress, and provider permissions belong to the downstream runtime -deployment and are not inferred from URI possession. +non-empty `argv` child and an optional unique `capability "demand"` child; it +never invokes a shell. Demand is denied by default, so a runtime that has not +declared the capability receives neither `Observe` nor an expectation to emit +`ObservationResult`. The executable is an external operator-trusted input; the +guest cannot choose or rewrite it. Environment, credentials, egress, and +provider permissions belong to the downstream runtime deployment and are not +inferred from URI possession. The descriptor selects `shared` or `perBinding` topology. `shared` starts one runtime for the exact `(catalog, scheme, profile generation)` and multiplexes @@ -432,105 +435,185 @@ bindings. `perBinding` starts one instance for each active binding. A per-binding runtime is the same protocol with one registration; topology does not select another lifecycle model. -Both modes speak the same versioned, newline-delimited JSON protocol over -supervisor-owned stdin/stdout: +Both modes speak the same ABI-3, newline-delimited JSON protocol over +supervisor-owned stdin/stdout. The following notation shows the normalized +messages. Message types and fields lower to camel case; the nested result is +tagged by `status`: ```text -host -> register { +Publication { + schemaId, mediaType, bytes, topics, facts? +} + +host -> Register { owner: { incarnation, claim }, bindingId, registration, uri, selector, carrierPath, previousDigest? } -host -> unregister { +host -> Unregister { owner: { incarnation, claim }, bindingId, registration } +host -> Observe { + owner: { incarnation, claim }, + bindingId, registration, demandWatermark +} -runtime -> publish { +runtime -> Publish { owner: { incarnation, claim }, bindingId, registration, - schemaId, mediaType, bytes, topics, facts?, observedAt? + ...Publication } -runtime -> health { +runtime -> Health { owner: { incarnation, claim }, bindingId?, registration?, state: starting|ready|degraded|failed, detail? } +runtime -> ObservationResult { + owner: { incarnation, claim }, + bindingId, registration, demandWatermark, + result: + { status: unchanged } + | { status: failed, diagnostic? } + | { status: published, publication: Publication } +} ``` +`Publication` is one reusable typed payload, not two similar publication +shapes. Periodic `Publish` flattens it into the existing ABI-3 base wire shape; +the published demand result carries the same value atomically with the outcome. +Neither message contains a host timestamp or runtime-computed digest. + The supervisor assigns a fresh directional owner claim to every runtime -incarnation. A new claim atomically fences the prior process and clears its -binding registrations. Every `publish` and binding-scoped `health` is accepted -only when both the owner claim and the host-generated registration token match -current state. `bindingId` is an opaque incarnation-scoped address, never the -binding name or URI. - -EOF ends the runtime protocol. The supervisor's existing process lifecycle is -the only shutdown and restart authority; there is no protocol `shutdown` -message. Observation belongs to the implementation, so there is no host -`reconcile` message. The runtime begins or resumes provider-native observation -after `register` and may use `previousDigest` to avoid redundant publication. - -Each encoded protocol line is at most 2 MiB, including the newline. `publish` -encodes `bytes` as a padded RFC 4648 base64 string; the decoded snapshot is -opaque. Selectors are at most 16 KiB when encoded as canonical compact JSON. -Decoded snapshot bytes are at most 1 MiB. Health `detail` is at most 16 KiB of -UTF-8. A publication has at most 32 ordered facts; keys are at most 128 bytes -and values at most 1 KiB of printable single-line UTF-8. A fact carries -`key` plus `before`, `after`, or both; explicit JSON null denotes absence. -These bounds are checked before allocation or decoding where the transport -permits and fail only the affected binding or runtime. st2 never truncates -canonical snapshot bytes, facts, or health text to satisfy a bound. - -The host rejects unknown bindings, stale owners or registrations, mismatched -schema or media type, unpublished topics, invalid messages, output after -unregister, and messages exceeding protocol bounds. A shared-runtime protocol -failure degrades every registered binding honestly but cannot publish across -schemes, profile generations, runtime incarnations, or binding registrations. +incarnation. A new claim fences the prior process and clears its binding +registrations. Every `Publish`, `ObservationResult`, binding-scoped `Health`, +and `Observe` dispatch is accepted or addressed only when owner claim, +`bindingId`, and host-generated registration token all match current state. +`bindingId` is an opaque incarnation-scoped address, never the binding name or +URI. + +EOF ends the runtime protocol. The supervisor's process lifecycle is the only +shutdown and restart authority; there is no protocol `Shutdown` message. The +runtime begins or resumes provider-native observation after `Register` and may +use `previousDigest` to avoid redundant periodic publication. `Observe` is a +level-triggered scheduling hint: it may pull an eligible observation forward, +but it is not provider reconciliation and cannot choose a provider mechanism, +reset polling cadence, backoff, cache, cursor, or rate-limit state, or authorize +a provider write. + +Each encoded protocol line is at most 2 MiB, including the newline. Snapshot +`bytes` use padded RFC 4648 base64 and decode to at most 1 MiB of opaque bytes. +Selectors are at most 16 KiB as canonical compact JSON. Health `detail` and a +failed-result `diagnostic` are each at most 16 KiB of UTF-8. A `Publication` +has at most 32 ordered facts; keys are at most 128 bytes and before/after +values are at most 1 KiB of printable single-line UTF-8. A fact carries `key` +plus `before`, `after`, or both; explicit JSON null denotes absence. Bounds are +checked before allocation or decoding where the transport permits and fail +only the affected binding or runtime. st2 does not truncate snapshot bytes, +facts, health text, or diagnostics to satisfy a bound. + +The host rejects unknown bindings, stale owners or registrations, zero demand +watermarks, mismatched schema or media type, unpublished topics, invalid facts +or messages, output after `Unregister`, and messages exceeding protocol bounds. +A shared-runtime protocol failure degrades every registered binding honestly +but cannot publish or settle demand across schemes, profile generations, +runtime incarnations, or binding registrations. + +For each exact active registration the supervisor has at most one `Observe` +dispatch in flight and one latest trailing demand watermark. Watermarks are +positive and monotonically increase within that registration. A matching +`ObservationResult` closes exactly the in-flight batch. Demand accepted while +that observation is in flight survives its result and coalesces into one +trailing dispatch. A registration replacement fences the old batch, and +provider-process or transport failure supplies failure evidence for it. +Backpressure leaves admitted, undispatched demand pending. No timeout, wall +clock, or normal polling cycle completes demand. + +The private durable request and receipt records are bounded to 64 KiB and carry +exact schema identities `st2.resource-observe-request.v1` and +`st2.resource-observe-receipt.v1`. One supervisor scope admits at most 256 +unresolved requests. Submission beyond that cap returns backpressure before +creating another request, and the supervisor scans no more than the cap. An +admitted request remains the durable retryable intent until a terminal receipt +is durably committed; in-memory enqueue and a nonterminal receipt are not +ownership transfer. +Terminal receipt failure keeps retryable state and leaves the request +available to a restarted supervisor. A terminal receipt is the durable +successor and only then permits request cleanup. + +Durable JSON receipt status values are camelCase. `accepted` and `backpressured` +are nonterminal. The terminal set is exactly `settledUnchanged`, +`settledChanged`, `settledFailed`, `absentBinding`, `staleGeneration`, and +`providerUnavailable`. Human CLI text renders multiword statuses in kebab-case. +`Unchanged` maps to `settledUnchanged`; `Failed` maps to `settledFailed` after +its provider diagnostic is normalized to a receipt-safe optional bounded value; +and an accepted `Published` maps to `settledChanged` with the host-computed +digest of its accepted bytes. No other receipt status carries a digest. + +A missing active binding reports `absentBinding`. An active observable binding +whose runtime did not declare `demand` also reports `absentBinding`, with the +explicit diagnostic `the profile runtime does not declare the demand +capability`. Only a client generation older than the resident supervisor is +`staleGeneration`; a newer client generation remains queued until supervisor +refresh. Provider failure reports `providerUnavailable`. A client wait bound +controls only how long that client waits and performs a final receipt read at +the deadline. Expiry or disconnect leaves admitted demand and any trailing +dispatch obligation intact. Any provider cursor, webhook delivery identity, redelivery, polling interval, -rate-limit state, and repair strategy remain runtime-private. +rate-limit state, conditional cache, backoff, and repair strategy remain +runtime-private. ## Snapshot publication (PROFILE-R14) -The resolver's contained carrier path is the observable snapshot path. A -successful `publish` follows one host-owned transaction: +The resolver's contained carrier is the observable snapshot authority. +Periodic `Publish` and demand-result `Published` enter one host-owned acceptance +transaction: ```text -validate binding + schema + topics + facts + size +validate current fences + Publication schema + topics + facts + bounds | v -write new bytes to contained sibling temporary file - | - v fsync file + atomic rename + parent sync +compute SHA-256 digest from accepted snapshot bytes | v -compute/record current sha256 digest and freshness +atomically replace the contained carrier and record current digest + freshness | - `-> equal digest: no invalidation - changed digest: apply binding selector and retain selected topics + facts - -The runtime never writes the carrier directly. Descriptor-relative no-follow -containment from the existing resolver contract applies to the temporary file, -final file, and replacement. Publication failure preserves the last proven -snapshot and marks freshness/health degraded. The first successful publication -is a state transition from unavailable to readable. If the publication carries -at least one selected topic, st2 schedules the same superseding invalidation as -for every later changed digest. This explicit first-publication wake prevents a + `-> equal digest: no state transition + changed digest: apply selector and retain selected topics + facts +``` + +The runtime never writes the carrier directly and never supplies its +authoritative digest. Existing descriptor-relative no-follow containment +applies to publication. Failure before acceptance preserves the last proven +snapshot and marks publication health degraded. + +The initial accepted publication changes the binding from unavailable to +readable. If it carries at least one selected topic, st2 schedules the same +superseding invalidation as for a later changed digest. This wake prevents a live agent from retaining an unreadable view after delayed startup or recovery. Equal publications and publications without selected topics remain silent. +`ObservationResult.Unchanged` closes demand without changing the carrier or +freshness. `ObservationResult.Failed` closes demand as failed and preserves the +last proven carrier. `ObservationResult.Published` is not settled until its +embedded `Publication` passes the same acceptance transaction as periodic +`Publish`. Once the snapshot and catch-up transaction commits, it settles as +`settledChanged` with the host-computed accepted digest even if subsequent +resync delivery emission fails. Its bytes, topics, and typed facts cannot +disagree with a separate settlement frame because no such frame exists. + Snapshot bytes are profile-defined and opaque to st2. `schemaId` and `mediaType` make the bytes interpretable without making st2 own their semantics. -The first contract has one snapshot per binding: no named facets, generation -manifest, profile event log, or host retention policy. +Each binding has one snapshot, not named facets, a generation manifest, a +profile event log, or a host retention history. ## Semantic invalidation and catch-up (PROFILE-R17..R20) -For every changed digest, including the first successful publication, the host -validates and preserves the runtime's ordered facts and intersects -`publish.topics` with the normalized binding selector. An empty intersection -updates the canonical snapshot and freshness without scheduling delivery. A -non-empty intersection updates this bounded per-binding state: +For every changed digest, including the initial accepted publication, the +common acceptance core preserves the `Publication`'s ordered facts and +intersects its topics with the normalized binding selector. An empty +intersection updates canonical state and freshness without scheduling delivery. +A non-empty intersection updates this bounded per-binding state: ```text current_snapshot_digest: Digest? @@ -554,26 +637,26 @@ body = { binding, snapshotDigest, topics, facts } Subjects are at most 96 Unicode scalars. Facts retain publication order and are included only whole; topic space is reserved before facts are admitted. If no -fact fits, a compatible bounded fallback remains. The durable body always -retains the complete bounded fact list. It contains no snapshot bytes, provider -payload, credential, URI, reason, or provider cursor. Existing event -deduplication, inbox storage, DING rendering, and supersession apply unchanged. -Multiple topics for one atomic publication produce one invalidation. +fact fits, a compatible bounded fallback remains. The durable body retains the +complete bounded fact list. It contains no snapshot bytes, provider payload, +credential, URI, reason, or provider cursor. Existing event deduplication, +inbox storage, DING rendering, and supersession apply unchanged. Multiple +topics for one atomic publication produce one invalidation. If delivery is unavailable, a relevant publication replaces the pending -selected topics and facts with the latest relevant publication and sets -`pending_relevant_change = true`. Later irrelevant publications may advance -`current_snapshot_digest` but do not clear that state. When delivery becomes -available, st2 emits at most one invalidation for the then-current digest with -the retained latest relevant fact envelope, and clears it only after event -ingress accepts the record. No transition backlog exists. This is -level-triggered current-state catch-up, not event replay. +selected topics and facts with that latest relevant semantic envelope and sets +`pending_relevant_change = true`. A later irrelevant publication may advance +`current_snapshot_digest` but does not clear the pending envelope. When +delivery becomes available, st2 emits at most one invalidation for the +then-current digest with the retained latest relevant topics and facts, and +clears pending state only after event ingress accepts the record. No transition +backlog exists. This is level-triggered current-state catch-up, not event replay. Health has separate descriptor, selector, runtime, observation, publication, and delivery stages. Every stage reports affected scheme and binding without including URI credentials or provider payloads. The last proven snapshot stays readable with explicit freshness when observation fails; failure never relabels -old bytes as a newly observed snapshot. +old bytes as newly observed state. ## Design questions diff --git a/docs/vrs/07-resource/spec.md b/docs/vrs/07-resource/spec.md index 4453c380..ac3a7730 100644 --- a/docs/vrs/07-resource/spec.md +++ b/docs/vrs/07-resource/spec.md @@ -90,6 +90,7 @@ Before it, 605 of 655 declarations on one live catalog had a ```text st2 resource ls [] [--json] st2 resource read [] [--json] +st2 resource refresh [] [--agent ] [--wait ] [--json] st2 resource add --uri --reason [--inactive-reason ] [--selector-json ] [--agent ] [--json] st2 resource remove [--agent ] [--json] st2 resource rename [--agent ] [--json] @@ -112,15 +113,25 @@ above is elided at `…`; the command prints it in full. The name column is aligned to the widest name; the checkout URI is elided in this document, not by `ls`, which prints every URI verbatim. -The read verbs project one agent's declared bindings; before them, bindings were -visible only through `st2 agents --json`. A read takes a leading identity and a -write takes `--agent `, both defaulting to the caller -(`--as` / `$ST_AGENT`); every verb inherits `--catalog`, `--root`, `--as`, and -`--host`. The write verbs perform read-modify-CAS-publish internally and emit a -stable `--json` receipt, so the caller never renders KDL. Full-catalog -validation, exact-target selection, compare-and-swap, and fail-closed -concurrent-change behavior are preserved, and a -binding-only change does not stop, replace, or relaunch healthy work +The read surfaces project one agent's declared bindings; before them, bindings +were visible only through `st2 agents --json`. `ls` takes an optional leading +identity. `read` and `refresh` treat one positional argument as a binding on the +caller and two as ` `; `refresh --agent ` is +the equivalent explicit-target form and cannot be combined with a leading +identity. Every form defaults through `--as` / `$ST_AGENT`, and every verb +inherits `--catalog`, `--root`, `--as`, and `--host`. + +`refresh` asks the resident observable profile runtime for one atomic current +observation. It never rewrites the Resource declaration. `--wait` bounds only +the client wait; timeout or disconnect leaves admitted demand queued. JSON and +durable receipt status values are camelCase; human CLI status text is +kebab-case. + +The writes take `--agent `, perform read-modify-CAS-publish +internally, and emit a stable `--json` receipt, so the caller never renders KDL. +Full-catalog validation, exact-target selection, compare-and-swap, and +fail-closed concurrent-change behavior are preserved, and a binding-only +change does not stop, replace, or relaunch healthy work ([R21](../requirements.md)). This is the fourth instance of the pattern in [`src/agent_author.rs`](../../../src/agent_author.rs), after streams, desired state, and presentation diff --git a/src/catalog.rs b/src/catalog.rs index 9be05283..f356d90e 100644 --- a/src/catalog.rs +++ b/src/catalog.rs @@ -24,15 +24,15 @@ use std::collections::BTreeSet; use std::path::{Component, Path, PathBuf}; -use anyhow::Context as _; -use agent_spec::profile::{ProfileClass, ResourceProfile, ResourceProfileRegistry}; #[cfg(feature = "wasm-resolver")] use agent_spec::profile::ProfileCapability; +use agent_spec::profile::{ProfileClass, ResourceProfile, ResourceProfileRegistry}; +use anyhow::Context as _; use kdl::KdlDocument; /// The catalog-level declaration, read from the catalog root. pub const CONFIG_FILE: &str = "catalog.kdl"; -/// One declared resource profile: `profile "" { wasm "" runtime { argv "..." } }`. +/// One declared resource profile with a closed direct runtime and optional `demand` capability. #[derive(Debug, Clone, PartialEq, Eq)] pub struct DeclaredProfile { /// The URI scheme this profile resolves. @@ -52,6 +52,8 @@ pub struct DeclaredProfile { #[derive(Debug, Clone, PartialEq, Eq)] pub struct DeclaredProfileRuntime { pub argv: Vec, + /// Opt in to demand-driven observation through one atomic observation result. + pub demand: bool, } /// What `/catalog.kdl` declares. An absent file leaves every field empty. @@ -124,10 +126,7 @@ pub fn parse(text: &str) -> anyhow::Result { "profile" => { let profile = parse_profile(node)?; if !seen_schemes.insert(profile.scheme.clone()) { - anyhow::bail!( - "profile '{}' declared more than once", - profile.scheme - ); + anyhow::bail!("profile '{}' declared more than once", profile.scheme); } config.profiles.push(profile); } @@ -186,36 +185,69 @@ fn parse_profile(node: &kdl::KdlNode) -> anyhow::Result { let runtime_children = child.children().ok_or_else(|| { anyhow::anyhow!("profile '{scheme}': runtime needs exactly one argv child") })?; - if runtime_children.nodes().len() != 1 - || runtime_children.nodes()[0].name().value() != "argv" - { - anyhow::bail!( - "profile '{scheme}': runtime accepts exactly one argv child" - ); - } - let argv_node = &runtime_children.nodes()[0]; - if argv_node.children().is_some() || argv_node.entries().is_empty() { - anyhow::bail!( - "profile '{scheme}': runtime argv needs one or more non-empty quoted arguments" - ); - } - let argv = argv_node - .entries() - .iter() - .map(|entry| { - (entry.name().is_none()) - .then(|| entry.value().as_string()) + let mut argv = None; + let mut demand = false; + for runtime_child in runtime_children.nodes() { + match runtime_child.name().value() { + "argv" => { + if argv.is_some() + || runtime_child.children().is_some() + || runtime_child.entries().is_empty() + { + anyhow::bail!( + "profile '{scheme}': runtime needs exactly one argv child with one or more arguments" + ); + } + argv = Some( + runtime_child + .entries() + .iter() + .map(|entry| { + (entry.name().is_none()) + .then(|| entry.value().as_string()) + .flatten() + .filter(|value| !value.is_empty()) + .map(str::to_owned) + }) + .collect::>>() + .ok_or_else(|| { + anyhow::anyhow!( + "profile '{scheme}': runtime argv accepts only non-empty quoted arguments" + ) + })?, + ); + } + "capability" => { + let entries = runtime_child.entries(); + let capability = (runtime_child.children().is_none() + && entries.len() == 1 + && entries[0].name().is_none()) + .then(|| entries[0].value().as_string()) .flatten() - .filter(|value| !value.is_empty()) - .map(str::to_owned) - }) - .collect::>>() - .ok_or_else(|| { - anyhow::anyhow!( - "profile '{scheme}': runtime argv accepts only non-empty quoted arguments" - ) - })?; - runtime = Some(DeclaredProfileRuntime { argv }); + .ok_or_else(|| { + anyhow::anyhow!( + "profile '{scheme}': runtime capability needs exactly one quoted value" + ) + })?; + anyhow::ensure!( + capability == "demand", + "profile '{scheme}': unknown runtime capability '{capability}'" + ); + anyhow::ensure!( + !demand, + "profile '{scheme}': runtime capability 'demand' is declared more than once" + ); + demand = true; + } + other => anyhow::bail!( + "profile '{scheme}': runtime field '{other}' is unknown (expected argv or capability)" + ), + } + } + let argv = argv.ok_or_else(|| { + anyhow::anyhow!("profile '{scheme}': runtime needs exactly one argv child") + })?; + runtime = Some(DeclaredProfileRuntime { argv, demand }); continue; } if child.children().is_some() { @@ -315,9 +347,7 @@ pub(crate) fn resolve_profile_module( let resolved = lexical_absolute(&catalog_root.join(expanded))?; let relative = resolved.strip_prefix(&catalog_root).with_context(|| { - format!( - "catalog-relative profile module escapes the catalog root: {declared}" - ) + format!("catalog-relative profile module escapes the catalog root: {declared}") })?; anyhow::ensure!( !relative.as_os_str().is_empty(), @@ -412,7 +442,6 @@ fn lexical_absolute(path: &Path) -> anyhow::Result { Ok(normalized) } - /// The session registry the CATALOG itself declares: `pty-root` if it declares one, else the native /// `/pty`. This is what `st2 env`/`st2 pty`/`st2 shell` hand to bus-aware tools, so those /// describe the catalog rather than whatever registry the caller happens to be standing in. @@ -485,15 +514,18 @@ pub fn passive_profiles( let refresh = registry.begin_refresh(); #[cfg(feature = "wasm-resolver")] { - config.profiles.iter().try_fold( - ResourceProfileRegistry::empty(), - |passive, declared| { + config + .profiles + .iter() + .try_fold(ResourceProfileRegistry::empty(), |passive, declared| { let observable = refresh .try_descriptor(&declared.scheme) .ok() .flatten() .is_some_and(|descriptor| { - descriptor.capabilities.contains(&ProfileCapability::Observe) + descriptor + .capabilities + .contains(&ProfileCapability::Observe) }); if observable { Ok(passive) @@ -505,8 +537,7 @@ pub fn passive_profiles( .clone(), )) } - }, - ) + }) } } @@ -516,7 +547,11 @@ fn validate_runtime_contracts( ) -> anyhow::Result<()> { #[cfg(not(feature = "wasm-resolver"))] { - if let Some(profile) = config.profiles.iter().find(|profile| profile.runtime.is_some()) { + if let Some(profile) = config + .profiles + .iter() + .find(|profile| profile.runtime.is_some()) + { anyhow::bail!( "profile '{}': observable runtime unavailable because st2 was built without the `wasm-resolver` feature", profile.scheme @@ -544,7 +579,9 @@ fn validate_runtime_contracts( } }; let observes = descriptor.as_ref().is_some_and(|descriptor| { - descriptor.capabilities.contains(&ProfileCapability::Observe) + descriptor + .capabilities + .contains(&ProfileCapability::Observe) }); match (observes, profile.runtime.is_some()) { (true, false) => anyhow::bail!( @@ -688,6 +725,7 @@ mod tests { wasm "observe.wasm" runtime { argv "github-resource-runtime" "pr" + capability "demand" } } "#, @@ -697,20 +735,26 @@ mod tests { config.profiles[0].runtime, Some(DeclaredProfileRuntime { argv: vec!["github-resource-runtime".into(), "pr".into()], + demand: true, }) ); for malformed in [ - r#"profile "dev.x" { wasm "x.wasm" runtime }"#, - r#"profile "dev.x" { wasm "x.wasm" runtime "shell" { argv "x" } }"#, - r#"profile "dev.x" { wasm "x.wasm" runtime { } }"#, - r#"profile "dev.x" { wasm "x.wasm" runtime { argv } }"#, - r#"profile "dev.x" { wasm "x.wasm" runtime { argv "" } }"#, - r#"profile "dev.x" { wasm "x.wasm" runtime { argv "x" argv "y" } }"#, - r#"profile "dev.x" { wasm "x.wasm" runtime { command "x" } }"#, - r#"profile "dev.x" { wasm "x.wasm" runtime { argv "x" } runtime { argv "y" } }"#, + r#"profile "dev.x" { wasm "x.wasm"; runtime }"#, + r#"profile "dev.x" { wasm "x.wasm"; runtime "shell" { argv "x" } }"#, + r#"profile "dev.x" { wasm "x.wasm"; runtime { } }"#, + r#"profile "dev.x" { wasm "x.wasm"; runtime { argv } }"#, + r#"profile "dev.x" { wasm "x.wasm"; runtime { argv "" } }"#, + r#"profile "dev.x" { wasm "x.wasm"; runtime { argv "x"; argv "y" } }"#, + r#"profile "dev.x" { wasm "x.wasm"; runtime { command "x" } }"#, + r#"profile "dev.x" { wasm "x.wasm"; runtime { argv "x"; capability "unknown" } }"#, + r#"profile "dev.x" { wasm "x.wasm"; runtime { argv "x"; capability "demand"; capability "demand" } }"#, + r#"profile "dev.x" { wasm "x.wasm"; runtime { argv "x" }; runtime { argv "y" } }"#, ] { assert!(parse(malformed).is_err(), "expected error for: {malformed}"); } + let without_demand = + parse(r#"profile "dev.x" { wasm "x.wasm"; runtime { argv "x" } }"#).unwrap(); + assert!(!without_demand.profiles[0].runtime.as_ref().unwrap().demand); } #[test] diff --git a/src/lib.rs b/src/lib.rs index 4d273870..cdefb3fd 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -43,6 +43,7 @@ pub mod pretrust; pub mod provider_session; pub mod reconcile; pub mod request; +pub mod resource_observe; pub mod resource_profile; pub mod resource_profile_supervisor; pub mod resync; diff --git a/src/main.rs b/src/main.rs index cd7d829f..df021f03 100644 --- a/src/main.rs +++ b/src/main.rs @@ -724,6 +724,24 @@ enum ResourceCmd { #[command(flatten)] ctx: MsgCtx, }, + /// Ask the resident profile runtime to observe one binding now and wait for exact evidence. + Refresh { + /// Binding name, or an agent selector when followed by a binding name. + first: String, + /// Binding name when the first positional selects the agent. + second: Option, + /// Exact target agent; defaults to --as / $ST_AGENT. + #[arg(long, conflicts_with = "second")] + agent: Option, + /// Client-only wait bound in seconds. Expiry never cancels or retracts queued demand. + #[arg(long, default_value_t = 30)] + wait: u64, + /// Emit the stable receipt (or timeout envelope) as JSON. + #[arg(long)] + json: bool, + #[command(flatten)] + ctx: MsgCtx, + }, /// Declare a Resource binding, or prove the identical binding already exists. Add { /// The agent-local binding name. @@ -1314,8 +1332,10 @@ fn dispatch(command: Command, catalog_path: Option<&std::path::Path>) -> Result< if !json { anyhow::bail!("`st2 catalog graph` v1 requires --json"); } - let graph = - st2::catalog_graph::snapshot(&catalog_arg(None)?, &host.unwrap_or_else(detect_host))?; + let graph = st2::catalog_graph::snapshot( + &catalog_arg(None)?, + &host.unwrap_or_else(detect_host), + )?; let complete = graph.complete; println!("{}", serde_json::to_string_pretty(&graph)?); if !complete { @@ -2089,10 +2109,9 @@ fn doctor_cmd(root: &Path, host: Option, require_supervisor: bool) -> Re { // The harness's own percent, never one st2 divided out of the operands beside it, // and never clamped: an overrun above 100 is exactly what this warns about. - if context - .used_percent - .is_some_and(|percent| percent >= st2::harness_context::HARNESS_CONTEXT_WARN_PERCENT) - { + if context.used_percent.is_some_and(|percent| { + percent >= st2::harness_context::HARNESS_CONTEXT_WARN_PERCENT + }) { report_advisory( &format!( "{bus_id} harness context at {}%", @@ -2118,8 +2137,7 @@ fn doctor_cmd(root: &Path, host: Option, require_supervisor: bool) -> Re } } if st2::driver_diagnostic::expected_for(spec) { - let diagnostic = - st2::driver_diagnostic::read(&st2::driver_diagnostic::path(dir)); + let diagnostic = st2::driver_diagnostic::read(&st2::driver_diagnostic::path(dir)); match &diagnostic { st2::driver_diagnostic::Observed::Failure(failure) => report_advisory( &format!( @@ -3361,6 +3379,19 @@ fn resource_bindings( selector: &str, host: &str, ) -> Result<(String, Vec)> { + with_resource_bindings_snapshot(root, selector, host, |identity, bindings| { + Ok((identity, bindings)) + }) +} + +/// Resolve one binding projection and let the caller finish consuming that exact catalog snapshot +/// before its shared authoring fence is released. +fn with_resource_bindings_snapshot( + root: &Path, + selector: &str, + host: &str, + consume: impl FnOnce(String, Vec) -> Result, +) -> Result { let _catalog_lock = st2::CatalogLock::shared(root) .context("acquire shared catalog-authoring lock for Resource bindings")?; let found = st2::discover_strict(root); @@ -3385,9 +3416,9 @@ fn resource_bindings( } else { exact }; - match matches.as_slice() { + let (identity, bindings) = match matches.as_slice() { [] => anyhow::bail!("no agent '{selector}' found in catalog {}", root.display()), - [spec] => Ok((spec.bus_id(host), spec.resources.clone())), + [spec] => (spec.bus_id(host), spec.resources.clone()), many => { let mut candidates = many .iter() @@ -3399,9 +3430,27 @@ fn resource_bindings( candidates.join(", ") ) } + }; + consume(identity, bindings) +} + +#[cfg(debug_assertions)] +fn resource_refresh_snapshot_checkpoint() { + let (Ok(ready), Ok(release)) = ( + std::env::var("ST2_TEST_RESOURCE_REFRESH_SNAPSHOT_READY"), + std::env::var("ST2_TEST_RESOURCE_REFRESH_SNAPSHOT_RELEASE"), + ) else { + return; + }; + let _ = std::fs::write(ready, b"ready"); + while !Path::new(&release).exists() { + std::thread::yield_now(); } } +#[cfg(not(debug_assertions))] +fn resource_refresh_snapshot_checkpoint() {} + fn resource_cmd(cmd: ResourceCmd) -> Result<()> { match cmd { ResourceCmd::Ls { @@ -3465,6 +3514,97 @@ fn resource_cmd(cmd: ResourceCmd) -> Result<()> { } Ok(()) } + ResourceCmd::Refresh { + first, + second, + agent, + wait, + json, + ctx, + } => { + let (root, host) = resolve_ctx(&ctx)?; + let (selector, name) = match second { + Some(name) => (first, name), + None => { + let selector = match agent { + Some(agent) => agent, + None => acting_id(&ctx)?, + }; + (selector, first) + } + }; + let (identity, request) = + with_resource_bindings_snapshot(&root, &selector, &host, |identity, bindings| { + bindings + .iter() + .find(|binding| binding.name() == name) + .with_context(|| { + format!("no resource binding '{name}' declared by {identity}") + })?; + resource_refresh_snapshot_checkpoint(); + let generation = st2::resource_observe::catalog_generation(&root)?; + let request = st2::resource_observe::ObserveRequest::new( + identity.clone(), + name.clone(), + generation, + None, + )?; + Ok((identity, request)) + })?; + let request_id = request.request_id.clone(); + let client = st2::resource_observe::submit_request(&root, &host, &request)?; + let waited = client.wait_for_terminal(Duration::from_secs(wait))?; + if waited.timed_out { + if json { + println!( + "{}", + serde_json::to_string(&serde_json::json!({ + "status": "timeout", + "requestId": request_id, + "queued": true, + "receipt": &waited.receipt, + }))? + ); + } + let last = waited + .receipt + .as_ref() + .map(|receipt| receipt.status.as_str()) + .unwrap_or("pending"); + anyhow::bail!( + "resource refresh for {identity} {name} exceeded the {wait}s client wait bound \ + (last status: {last}); the request remains queued and may still settle" + ); + } + let receipt = waited + .receipt + .context("Resource observation ended without a receipt")?; + if json { + println!("{}", serde_json::to_string(&receipt)?); + } + if !receipt.status.is_success() { + anyhow::bail!( + "resource refresh for {identity} {name}: {}{}", + receipt.status.as_str(), + receipt + .diagnostic + .as_deref() + .map(|detail| format!(": {detail}")) + .unwrap_or_default() + ); + } + if !json { + let digest = receipt + .digest + .map(|digest| format!(" ({digest})")) + .unwrap_or_default(); + println!( + "resource {name} on {identity}: {}{digest}", + receipt.status.as_str() + ); + } + Ok(()) + } ResourceCmd::Add { name, uri, diff --git a/src/metrics.rs b/src/metrics.rs index 4e075c75..d75af725 100644 --- a/src/metrics.rs +++ b/src/metrics.rs @@ -95,6 +95,27 @@ static DRIVER_DIAGNOSTICS: LazyLock> = LazyLock::new(|| { .with_unit("1") .build() }); +static RESOURCE_OBSERVE_REQUESTS: LazyLock> = LazyLock::new(|| { + METER + .u64_counter("resource_observe_requests_total") + .with_description("Demand-observation request lifecycle transitions by bounded outcome") + .with_unit("1") + .build() +}); +static RESOURCE_OBSERVE_DISPATCH: LazyLock> = LazyLock::new(|| { + METER + .f64_histogram("resource_observe_dispatch_seconds") + .with_description("Latency from request publication to provider-runtime dispatch") + .with_unit("s") + .build() +}); +static RESOURCE_OBSERVE_SETTLE: LazyLock> = LazyLock::new(|| { + METER + .f64_histogram("resource_observe_settle_seconds") + .with_description("Latency from provider-runtime dispatch to durable result receipt") + .with_unit("s") + .build() +}); /// One reconcile pass finished. `failed` = the pass collected errors. pub fn record_reconcile_pass(duration: Duration, failed: bool) { @@ -104,7 +125,10 @@ pub fn record_reconcile_pass(duration: Duration, failed: bool) { RECONCILE_PASS_DURATION.record(duration.as_secs_f64(), &[]); RECONCILE_PASSES.add( 1, - &[opentelemetry::KeyValue::new("result", if failed { "fail" } else { "pass" })], + &[opentelemetry::KeyValue::new( + "result", + if failed { "fail" } else { "pass" }, + )], ); } @@ -147,7 +171,10 @@ pub fn record_message_delivery(failed: bool) { } MESSAGE_DELIVERIES.add( 1, - &[opentelemetry::KeyValue::new("result", if failed { "fail" } else { "pass" })], + &[opentelemetry::KeyValue::new( + "result", + if failed { "fail" } else { "pass" }, + )], ); } @@ -175,6 +202,32 @@ pub fn record_driver_diagnostic( &driver_diagnostic_attributes(stage, reason, source, support, recovered), ); } +pub fn record_resource_observe_request(outcome: &'static str) { + if !enabled() { + return; + } + RESOURCE_OBSERVE_REQUESTS.add( + 1, + &[opentelemetry::KeyValue::new( + "outcome", + normalize_resource_observe_outcome(outcome), + )], + ); +} + +pub fn record_resource_observe_dispatch(duration: Duration) { + if !enabled() { + return; + } + RESOURCE_OBSERVE_DISPATCH.record(duration.as_secs_f64(), &[]); +} + +pub fn record_resource_observe_settle(duration: Duration) { + if !enabled() { + return; + } + RESOURCE_OBSERVE_SETTLE.record(duration.as_secs_f64(), &[]); +} fn driver_diagnostic_attributes( stage: Stage, @@ -191,6 +244,19 @@ fn driver_diagnostic_attributes( opentelemetry::KeyValue::new("outcome", if recovered { "recovery" } else { "failure" }), ] } +fn normalize_resource_observe_outcome(outcome: &str) -> &'static str { + match outcome { + "accepted" => "accepted", + "backpressured" => "backpressured", + "settledUnchanged" => "settledUnchanged", + "settledChanged" => "settledChanged", + "settledFailed" => "settledFailed", + "absentBinding" => "absentBinding", + "staleGeneration" => "staleGeneration", + "providerUnavailable" => "providerUnavailable", + _ => "other", + } +} /// The bounded Claude hook-event vocabulary st2 applies; anything else is `other`. fn normalize_hook_event(event: &str) -> &'static str { @@ -225,6 +291,9 @@ mod tests { record_hook_invocation("claude-observe", "SomethingUnheardOf"); record_message_delivery(true); record_crash_loop(); + record_resource_observe_request("accepted"); + record_resource_observe_dispatch(Duration::from_millis(1)); + record_resource_observe_settle(Duration::from_millis(1)); assert!(!enabled()); record_driver_diagnostic( @@ -241,6 +310,18 @@ mod tests { assert_eq!(normalize_hook_event("SessionStart"), "SessionStart"); assert_eq!(normalize_hook_event("TotallyNewEvent"), "other"); } + #[test] + fn resource_observe_outcomes_use_the_bounded_wire_vocabulary() { + assert_eq!( + normalize_resource_observe_outcome("settledUnchanged"), + "settledUnchanged" + ); + assert_eq!( + normalize_resource_observe_outcome("settled-unchanged"), + "other" + ); + assert_eq!(normalize_resource_observe_outcome("h.worker"), "other"); + } #[test] fn driver_diagnostic_metric_attributes_are_exactly_the_bounded_axes() { @@ -251,7 +332,10 @@ mod tests { Support::Supported, true, ); - let keys: Vec<&str> = attributes.iter().map(|attribute| attribute.key.as_str()).collect(); + let keys: Vec<&str> = attributes + .iter() + .map(|attribute| attribute.key.as_str()) + .collect(); assert_eq!(keys, ["stage", "reason", "source", "support", "outcome"]); let rendered = format!("{attributes:?}"); for forbidden in ["1.18.19", "h.worker", "ses_", "msg_"] { diff --git a/src/park.rs b/src/park.rs index 1456edc3..13310e9e 100644 --- a/src/park.rs +++ b/src/park.rs @@ -56,7 +56,11 @@ impl SupervisorScope { Self::in_state_root(&crate::run::state_root(), catalog_root, host) } - fn in_state_root(state_root: &Path, catalog_root: &Path, host: &str) -> anyhow::Result { + pub(crate) fn in_state_root( + state_root: &Path, + catalog_root: &Path, + host: &str, + ) -> anyhow::Result { let catalog_root = catalog_root.canonicalize().with_context(|| { format!("canonicalize supervisor catalog {}", catalog_root.display()) })?; @@ -69,6 +73,9 @@ impl SupervisorScope { root: state_root.join("st2/supervisors").join(scope_id), }) } + pub(crate) fn root(&self) -> &Path { + &self.root + } pub fn park_dir(&self) -> PathBuf { self.root.join("parked") @@ -77,6 +84,13 @@ impl SupervisorScope { pub fn unpark_request_dir(&self) -> PathBuf { self.root.join("unpark") } + pub(crate) fn observe_request_dir(&self) -> PathBuf { + self.root.join("observe-requests") + } + + pub(crate) fn observe_receipt_dir(&self) -> PathBuf { + self.root.join("observe-receipts") + } pub(crate) fn stream_owner_binding_path(&self) -> PathBuf { self.root.join("stream-owner.json") diff --git a/src/resource_observe.rs b/src/resource_observe.rs new file mode 100644 index 00000000..0b00e839 --- /dev/null +++ b/src/resource_observe.rs @@ -0,0 +1,939 @@ +//! Private, scope-owned request/receipt channel for demand observation. +//! +//! A CLI writes one bounded request record; the resident Resource Profile supervisor alone writes +//! its receipt. Both directions fail toward missing evidence. Atomic rename prevents partial JSON +//! from becoming control input, while deliberately omitting fsync means power loss can require a +//! retry but can never manufacture success. + +use std::fs::{self, OpenOptions}; +use std::io::{Read as _, Write as _}; +use std::os::fd::AsRawFd as _; +use std::os::unix::fs::{OpenOptionsExt as _, PermissionsExt as _}; +use std::path::{Component, Path, PathBuf}; +use std::sync::atomic::{AtomicU64, Ordering}; +use std::sync::mpsc::{self, Receiver}; +use std::time::{Duration, Instant, SystemTime}; + +use anyhow::Context as _; +use notify::RecommendedWatcher; +use serde::{Deserialize, Serialize}; + +use crate::park::SupervisorScope; +use crate::resource_profile::{BindingId, RegistrationToken, RuntimeOwner, SnapshotDigest}; + +pub const OBSERVE_REQUEST_SCHEMA: &str = "st2.resource-observe-request.v1"; +pub const OBSERVE_RECEIPT_SCHEMA: &str = "st2.resource-observe-receipt.v1"; +pub const MAX_PENDING_OBSERVE_REQUESTS: usize = 256; +pub const MAX_OBSERVE_RECEIPTS: usize = 256; +const MAX_CONTROL_RECORD_BYTES: u64 = 64 * 1024; +const MAX_CONTROL_TEXT_BYTES: usize = 16 * 1024; +const MAX_REQUEST_ID_BYTES: usize = 128; +const SCOPE_MODE: u32 = 0o700; +const RECORD_MODE: u32 = 0o600; +static REQUEST_SEQUENCE: AtomicU64 = AtomicU64::new(1); + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct ObserveAdmissionBackpressure { + limit: usize, +} + +impl ObserveAdmissionBackpressure { + pub fn limit(self) -> usize { + self.limit + } +} + +impl std::fmt::Display for ObserveAdmissionBackpressure { + fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!( + formatter, + "durable Resource observation backlog is full (limit {})", + self.limit + ) + } +} + +impl std::error::Error for ObserveAdmissionBackpressure {} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase", deny_unknown_fields)] +pub struct ObserveRequest { + pub schema: String, + pub request_id: String, + pub recipient: String, + pub binding: String, + #[serde(skip_serializing_if = "Option::is_none")] + pub expected_catalog_generation: Option, + /// Optional client fence against the host's currently accepted snapshot. + #[serde(skip_serializing_if = "Option::is_none")] + pub expected_snapshot_digest: Option, + pub requested_at: String, +} + +impl ObserveRequest { + pub fn new( + recipient: String, + binding: String, + expected_catalog_generation: Option, + expected_snapshot_digest: Option, + ) -> anyhow::Result { + validate_control_text("recipient", &recipient)?; + validate_control_text("binding", &binding)?; + let request_id = format!( + "{}-{}-{:x}", + crate::message::now_ms(), + std::process::id(), + REQUEST_SEQUENCE.fetch_add(1, Ordering::Relaxed) + ); + Ok(Self { + schema: OBSERVE_REQUEST_SCHEMA.to_owned(), + request_id, + recipient, + binding, + expected_catalog_generation, + expected_snapshot_digest, + requested_at: crate::exec_backend::rfc3339_utc(SystemTime::now())?, + }) + } + + pub(crate) fn validate(&self) -> anyhow::Result<()> { + anyhow::ensure!( + self.schema == OBSERVE_REQUEST_SCHEMA, + "unsupported observe request schema {:?}", + self.schema + ); + validate_request_id(&self.request_id)?; + validate_control_text("recipient", &self.recipient)?; + validate_control_text("binding", &self.binding)?; + validate_control_text("requestedAt", &self.requested_at) + } + + pub(crate) fn stable_key(&self) -> String { + format!("{}\0{}", self.recipient, self.binding) + } +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase", deny_unknown_fields)] +pub struct ObservationAuthority { + pub owner: RuntimeOwner, + pub binding_id: BindingId, + pub registration: RegistrationToken, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub enum ObserveReceiptStatus { + Accepted, + Backpressured, + SettledUnchanged, + SettledChanged, + SettledFailed, + AbsentBinding, + StaleGeneration, + ProviderUnavailable, +} + +impl ObserveReceiptStatus { + pub fn is_terminal(self) -> bool { + !matches!(self, Self::Accepted | Self::Backpressured) + } + + pub fn is_success(self) -> bool { + matches!(self, Self::SettledUnchanged | Self::SettledChanged) + } + + pub fn as_str(self) -> &'static str { + match self { + Self::Accepted => "accepted", + Self::Backpressured => "backpressured", + Self::SettledUnchanged => "settled-unchanged", + Self::SettledChanged => "settled-changed", + Self::SettledFailed => "settled-failed", + Self::AbsentBinding => "absent-binding", + Self::StaleGeneration => "stale-generation", + Self::ProviderUnavailable => "provider-unavailable", + } + } + + pub fn wire_str(self) -> &'static str { + match self { + Self::Accepted => "accepted", + Self::Backpressured => "backpressured", + Self::SettledUnchanged => "settledUnchanged", + Self::SettledChanged => "settledChanged", + Self::SettledFailed => "settledFailed", + Self::AbsentBinding => "absentBinding", + Self::StaleGeneration => "staleGeneration", + Self::ProviderUnavailable => "providerUnavailable", + } + } +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase", deny_unknown_fields)] +pub struct ObserveReceipt { + pub schema: String, + pub request_id: String, + pub recipient: String, + pub binding: String, + pub status: ObserveReceiptStatus, + #[serde(skip_serializing_if = "Option::is_none")] + pub authority: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub demand_watermark: Option, + /// Present only for `SettledChanged`; computed by the host from accepted publication bytes. + #[serde(skip_serializing_if = "Option::is_none")] + pub digest: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub diagnostic: Option, + pub updated_at: String, +} + +impl ObserveReceipt { + pub(crate) fn new( + request: &ObserveRequest, + status: ObserveReceiptStatus, + authority: Option, + demand_watermark: Option, + digest: Option, + diagnostic: Option, + ) -> anyhow::Result { + let diagnostic = normalize_diagnostic(diagnostic); + Ok(Self { + schema: OBSERVE_RECEIPT_SCHEMA.to_owned(), + request_id: request.request_id.clone(), + recipient: request.recipient.clone(), + binding: request.binding.clone(), + status, + authority, + demand_watermark, + digest, + diagnostic, + updated_at: crate::exec_backend::rfc3339_utc(SystemTime::now())?, + }) + } + + fn validate(&self) -> anyhow::Result<()> { + anyhow::ensure!( + self.schema == OBSERVE_RECEIPT_SCHEMA, + "unsupported observe receipt schema {:?}", + self.schema + ); + validate_request_id(&self.request_id)?; + validate_control_text("recipient", &self.recipient)?; + validate_control_text("binding", &self.binding)?; + validate_control_text("updatedAt", &self.updated_at)?; + if let Some(diagnostic) = self.diagnostic.as_deref() { + validate_control_text("diagnostic", diagnostic)?; + } + let dispatched = matches!( + self.status, + ObserveReceiptStatus::Accepted + | ObserveReceiptStatus::Backpressured + | ObserveReceiptStatus::SettledUnchanged + | ObserveReceiptStatus::SettledChanged + | ObserveReceiptStatus::SettledFailed + ); + match (&self.authority, self.demand_watermark) { + (Some(_), Some(watermark)) if watermark > 0 => {} + (None, None) if !dispatched => {} + (None, None) => anyhow::bail!("receipt status requires authority and demand watermark"), + _ => anyhow::bail!( + "receipt authority and positive demand watermark must be present together" + ), + } + match self.status { + ObserveReceiptStatus::SettledChanged => { + anyhow::ensure!( + self.digest.is_some(), + "settled-changed receipt requires the accepted publication digest" + ); + } + _ => anyhow::ensure!( + self.digest.is_none(), + "only a settled-changed receipt may carry a publication digest" + ), + } + Ok(()) + } +} + +#[derive(Debug)] +pub struct ObserveWait { + pub receipt: Option, + pub timed_out: bool, +} + +pub struct ObserveClient { + receipt_path: PathBuf, + wake: Receiver<()>, + _watcher: Option, +} + +impl ObserveClient { + pub fn wait_for_terminal(self, bound: Duration) -> anyhow::Result { + let deadline = Instant::now() + .checked_add(bound) + .context("observe wait bound is too large")?; + let mut last = None; + loop { + if let Some(receipt) = read_receipt_path(&self.receipt_path)? { + let terminal = receipt.status.is_terminal(); + last = Some(receipt); + if terminal { + return Ok(ObserveWait { + receipt: last, + timed_out: false, + }); + } + } + let now = Instant::now(); + if now >= deadline { + return finish_wait_at_timeout(&self.receipt_path, last); + } + let remaining = deadline.saturating_duration_since(now); + if self._watcher.is_some() { + match self.wake.recv_timeout(remaining) { + Ok(()) => {} + Err(mpsc::RecvTimeoutError::Timeout) => { + return finish_wait_at_timeout(&self.receipt_path, last); + } + Err(mpsc::RecvTimeoutError::Disconnected) => { + std::thread::sleep(remaining.min(Duration::from_millis(25))); + } + } + } else { + std::thread::sleep(remaining.min(Duration::from_millis(25))); + } + } + } +} + +pub fn catalog_generation(catalog_root: &Path) -> anyhow::Result> { + crate::catalog_lock::read_generation_token(catalog_root) +} + +/// Prove a live resident supervisor, install the receipt watch, then atomically publish the request. +/// Installing the watch before the write and always reading once before waiting closes the startup +/// race without any correctness timer. +pub fn submit_request( + catalog_root: &Path, + host: &str, + request: &ObserveRequest, +) -> anyhow::Result { + request.validate()?; + crate::event::current_stream_owner_incarnation(catalog_root, host) + .context("no live Resource Profile supervisor")?; + let scope = SupervisorScope::current(catalog_root, host)?; + prepare_scope(&scope)?; + let receipt_dir = scope.observe_receipt_dir(); + let request_dir = scope.observe_request_dir(); + let (wake_tx, wake) = mpsc::channel(); + let watcher = crate::watch::watch_recursive_mutations(&receipt_dir, wake_tx); + let receipt_path = receipt_path(&receipt_dir, &request.request_id)?; + let request_path = request_path(&request_dir, &request.request_id)?; + let _lock = lock_request_scope(&request_dir)?; + prune_request_temp_files(&request_dir)?; + if !request_path.exists() && durable_request_capacity_is_full(&request_dir)? { + return Err(ObserveAdmissionBackpressure { + limit: MAX_PENDING_OBSERVE_REQUESTS, + } + .into()); + } + write_json_atomically_no_fsync(&request_path, request)?; + Ok(ObserveClient { + receipt_path, + wake, + _watcher: watcher, + }) +} + +pub(crate) fn prepare_scope(scope: &SupervisorScope) -> anyhow::Result<()> { + ensure_private_directory(scope.root())?; + ensure_private_directory(&scope.observe_request_dir())?; + ensure_private_directory(&scope.observe_receipt_dir())?; + let request_dir = scope.observe_request_dir(); + let _lock = lock_request_scope(&request_dir)?; + prune_request_temp_files(&request_dir) +} + +#[derive(Debug)] +pub(crate) struct PendingRequestRecord { + pub request: ObserveRequest, + pub modified_at: SystemTime, + pub path: PathBuf, +} + +pub(crate) fn scan_requests(dir: &Path) -> (Vec, Vec) { + let mut records = Vec::new(); + let mut errors = Vec::new(); + { + let _lock = match lock_request_scope(dir) { + Ok(lock) => lock, + Err(error) => { + errors.push(format!( + "locking observe requests {}: {error:#}", + dir.display() + )); + return (records, errors); + } + }; + if let Err(error) = prune_request_temp_files(dir) { + errors.push(format!( + "pruning observe request temps {}: {error:#}", + dir.display() + )); + return (records, errors); + } + } + let entries = match fs::read_dir(dir) { + Ok(entries) => entries, + Err(error) if error.kind() == std::io::ErrorKind::NotFound => return (records, errors), + Err(error) => { + errors.push(format!( + "listing observe requests {}: {error}", + dir.display() + )); + return (records, errors); + } + }; + for (entry, request_id) in entries + .flatten() + .filter_map(|entry| { + let name = entry.file_name(); + let request_id = final_request_id(name.to_str()?)?.to_owned(); + Some((entry, request_id)) + }) + .take(MAX_PENDING_OBSERVE_REQUESTS) + { + let name = entry.file_name(); + let name = name.to_string_lossy(); + let path = entry.path(); + let parsed = (|| -> anyhow::Result { + let metadata = entry.metadata()?; + anyhow::ensure!(metadata.is_file(), "request record is not a regular file"); + let bytes = read_bounded_regular(&path)?; + let request: ObserveRequest = serde_json::from_slice(&bytes)?; + request.validate()?; + anyhow::ensure!( + request.request_id == request_id, + "request id does not match final basename" + ); + Ok(PendingRequestRecord { + request, + modified_at: metadata.modified().unwrap_or(SystemTime::now()), + path: path.clone(), + }) + })(); + match parsed { + Ok(record) => records.push(record), + Err(error) => { + errors.push(format!("invalid observe request {name:?}: {error:#}")); + let _ = remove_request(&path); + } + } + } + records.sort_by(|left, right| left.request.request_id.cmp(&right.request.request_id)); + (records, errors) +} + +pub(crate) fn remove_request(path: &Path) -> anyhow::Result<()> { + let request_dir = path + .parent() + .ok_or_else(|| anyhow::anyhow!("{} has no request directory", path.display()))?; + let _lock = lock_request_scope(request_dir)?; + match fs::remove_file(path) { + Ok(()) => Ok(()), + Err(error) if error.kind() == std::io::ErrorKind::NotFound => Ok(()), + Err(error) => Err(error).with_context(|| format!("remove {}", path.display())), + } +} + +pub(crate) fn write_receipt(dir: &Path, receipt: &ObserveReceipt) -> anyhow::Result<()> { + receipt.validate()?; + let path = receipt_path(dir, &receipt.request_id)?; + write_json_atomically_no_fsync(&path, receipt) +} + +pub fn read_receipt(dir: &Path, request_id: &str) -> anyhow::Result> { + read_receipt_path(&receipt_path(dir, request_id)?) +} + +fn read_receipt_path(path: &Path) -> anyhow::Result> { + let bytes = match read_bounded_regular(path) { + Ok(bytes) => bytes, + Err(error) + if error + .downcast_ref::() + .is_some_and(|error| error.kind() == std::io::ErrorKind::NotFound) => + { + return Ok(None); + } + Err(error) => return Err(error), + }; + let receipt: ObserveReceipt = serde_json::from_slice(&bytes) + .with_context(|| format!("decode observe receipt {}", path.display()))?; + receipt.validate()?; + Ok(Some(receipt)) +} + +pub(crate) fn prune_terminal_receipts(dir: &Path) -> Vec { + let mut terminal = Vec::new(); + let mut errors = Vec::new(); + let Ok(entries) = fs::read_dir(dir) else { + return errors; + }; + for entry in entries.flatten() { + let name = entry.file_name(); + let Some(name) = name.to_str() else { continue }; + if final_request_id(name).is_none() { + continue; + } + let path = entry.path(); + match read_receipt_path(&path) { + Ok(Some(receipt)) if receipt.status.is_terminal() => { + let modified = entry + .metadata() + .and_then(|metadata| metadata.modified()) + .unwrap_or(SystemTime::UNIX_EPOCH); + terminal.push((modified, path)); + } + Ok(_) => {} + Err(error) => errors.push(format!("inspect observe receipt {name:?}: {error:#}")), + } + } + terminal.sort_by_key(|(modified, _)| *modified); + let remove_count = terminal.len().saturating_sub(MAX_OBSERVE_RECEIPTS); + for (_, path) in terminal.into_iter().take(remove_count) { + if let Err(error) = fs::remove_file(&path) { + errors.push(format!("prune observe receipt {}: {error}", path.display())); + } + } + errors +} + +fn request_path(dir: &Path, request_id: &str) -> anyhow::Result { + validate_request_id(request_id)?; + Ok(dir.join(format!("{request_id}.json"))) +} + +fn receipt_path(dir: &Path, request_id: &str) -> anyhow::Result { + validate_request_id(request_id)?; + Ok(dir.join(format!("{request_id}.json"))) +} + +fn final_request_id(name: &str) -> Option<&str> { + let request_id = name.strip_suffix(".json")?; + validate_request_id(request_id).ok()?; + Some(request_id) +} + +fn validate_request_id(request_id: &str) -> anyhow::Result<()> { + let mut components = Path::new(request_id).components(); + let plain = matches!( + (components.next(), components.next()), + (Some(Component::Normal(_)), None) + ); + anyhow::ensure!( + plain + && !request_id.starts_with('.') + && request_id.len() <= MAX_REQUEST_ID_BYTES + && request_id + .bytes() + .all(|byte| byte.is_ascii_alphanumeric() || matches!(byte, b'-' | b'_')), + "{request_id:?} is not a valid observe request id" + ); + Ok(()) +} + +fn validate_control_text(field: &str, value: &str) -> anyhow::Result<()> { + anyhow::ensure!(!value.is_empty(), "{field} must not be empty"); + anyhow::ensure!( + value.len() <= MAX_CONTROL_TEXT_BYTES, + "{field} exceeds {MAX_CONTROL_TEXT_BYTES} bytes" + ); + anyhow::ensure!( + !value.bytes().any(|byte| byte == 0), + "{field} contains a NUL byte" + ); + Ok(()) +} + +fn normalize_diagnostic(diagnostic: Option) -> Option { + let mut diagnostic = diagnostic?; + if diagnostic.is_empty() { + return None; + } + if diagnostic.contains('\0') { + diagnostic = diagnostic.replace('\0', "\u{fffd}"); + } + if diagnostic.len() > MAX_CONTROL_TEXT_BYTES { + let mut end = MAX_CONTROL_TEXT_BYTES; + while !diagnostic.is_char_boundary(end) { + end -= 1; + } + diagnostic.truncate(end); + } + (!diagnostic.is_empty()).then_some(diagnostic) +} + +fn durable_request_capacity_is_full(dir: &Path) -> anyhow::Result { + let entries = fs::read_dir(dir).with_context(|| format!("list {}", dir.display()))?; + Ok(entries + .flatten() + .filter(|entry| { + entry + .file_name() + .to_str() + .and_then(final_request_id) + .is_some() + }) + .take(MAX_PENDING_OBSERVE_REQUESTS) + .count() + >= MAX_PENDING_OBSERVE_REQUESTS) +} + +fn prune_request_temp_files(dir: &Path) -> anyhow::Result<()> { + let entries = fs::read_dir(dir).with_context(|| format!("list {}", dir.display()))?; + for entry in entries.flatten() { + let name = entry.file_name(); + let Some(name) = name.to_str() else { continue }; + if name.starts_with(".observe-") { + fs::remove_file(entry.path()) + .with_context(|| format!("remove stale observe request temp {name:?}"))?; + } + } + Ok(()) +} + +fn lock_request_scope(request_dir: &Path) -> anyhow::Result { + let scope_root = request_dir.parent().ok_or_else(|| { + anyhow::anyhow!( + "observe request directory {} has no scope root", + request_dir.display() + ) + })?; + ensure_private_directory(scope_root)?; + let lock_path = scope_root.join(".observe.lock"); + let lock = OpenOptions::new() + .create(true) + .read(true) + .write(true) + .mode(RECORD_MODE) + .open(&lock_path) + .with_context(|| format!("open observe scope lock {}", lock_path.display()))?; + lock.set_permissions(fs::Permissions::from_mode(RECORD_MODE)) + .with_context(|| format!("set private permissions on {}", lock_path.display()))?; + let result = unsafe { libc::flock(lock.as_raw_fd(), libc::LOCK_EX) }; + if result != 0 { + return Err(std::io::Error::last_os_error()) + .with_context(|| format!("lock observe scope {}", scope_root.display())); + } + Ok(lock) +} + +fn finish_wait_at_timeout( + receipt_path: &Path, + last: Option, +) -> anyhow::Result { + test_observe_wait_timeout_checkpoint(); + let latest = read_receipt_path(receipt_path)?; + let receipt = latest.or(last); + let timed_out = !receipt + .as_ref() + .is_some_and(|receipt| receipt.status.is_terminal()); + Ok(ObserveWait { receipt, timed_out }) +} + +#[cfg(debug_assertions)] +fn test_observe_wait_timeout_checkpoint() { + let (Ok(ready), Ok(release)) = ( + std::env::var("ST2_TEST_OBSERVE_WAIT_TIMEOUT_READY"), + std::env::var("ST2_TEST_OBSERVE_WAIT_TIMEOUT_RELEASE"), + ) else { + return; + }; + let _ = fs::write(ready, b"ready"); + while !Path::new(&release).is_file() { + std::thread::yield_now(); + } +} + +#[cfg(not(debug_assertions))] +fn test_observe_wait_timeout_checkpoint() {} + +fn ensure_private_directory(path: &Path) -> anyhow::Result<()> { + fs::create_dir_all(path).with_context(|| format!("create {}", path.display()))?; + fs::set_permissions(path, fs::Permissions::from_mode(SCOPE_MODE)) + .with_context(|| format!("set private permissions on {}", path.display())) +} + +fn write_json_atomically_no_fsync(path: &Path, value: &T) -> anyhow::Result<()> { + let parent = path + .parent() + .ok_or_else(|| anyhow::anyhow!("{} has no parent", path.display()))?; + ensure_private_directory(parent)?; + let mut bytes = serde_json::to_vec(value)?; + bytes.push(b'\n'); + anyhow::ensure!( + bytes.len() as u64 <= MAX_CONTROL_RECORD_BYTES, + "control record exceeds {MAX_CONTROL_RECORD_BYTES} bytes" + ); + let mut temp = tempfile::Builder::new() + .prefix(".observe-") + .tempfile_in(parent)?; + temp.as_file() + .set_permissions(fs::Permissions::from_mode(RECORD_MODE))?; + temp.write_all(&bytes)?; + temp.flush()?; + temp.persist(path) + .map_err(|error| error.error) + .with_context(|| format!("publish {}", path.display()))?; + Ok(()) +} + +fn read_bounded_regular(path: &Path) -> anyhow::Result> { + let file = OpenOptions::new() + .read(true) + .custom_flags(libc::O_NOFOLLOW | libc::O_CLOEXEC | libc::O_NONBLOCK) + .open(path)?; + let metadata = file.metadata()?; + anyhow::ensure!( + metadata.is_file(), + "{} is not a regular file", + path.display() + ); + anyhow::ensure!( + metadata.len() <= MAX_CONTROL_RECORD_BYTES, + "{} exceeds {MAX_CONTROL_RECORD_BYTES} bytes", + path.display() + ); + let mut bytes = Vec::with_capacity(metadata.len() as usize); + file.take(MAX_CONTROL_RECORD_BYTES + 1) + .read_to_end(&mut bytes)?; + anyhow::ensure!( + bytes.len() as u64 <= MAX_CONTROL_RECORD_BYTES, + "{} grew beyond {MAX_CONTROL_RECORD_BYTES} bytes", + path.display() + ); + Ok(bytes) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn atomic_records_are_strict_private_and_round_trip() { + let temp = tempfile::tempdir().unwrap(); + let scope = SupervisorScope::in_state_root(temp.path(), temp.path(), "host").unwrap(); + prepare_scope(&scope).unwrap(); + let request = + ObserveRequest::new("h.worker".into(), "queue".into(), Some(9), None).unwrap(); + let path = request_path(&scope.observe_request_dir(), &request.request_id).unwrap(); + write_json_atomically_no_fsync(&path, &request).unwrap(); + let (records, errors) = scan_requests(&scope.observe_request_dir()); + assert!(errors.is_empty()); + assert_eq!(records.len(), 1); + assert_eq!(records[0].request, request); + assert_eq!( + fs::metadata(scope.root()).unwrap().permissions().mode() & 0o777, + SCOPE_MODE + ); + assert_eq!( + fs::metadata(path).unwrap().permissions().mode() & 0o777, + RECORD_MODE + ); + } + + #[test] + fn final_basename_filter_prunes_crash_temps_without_starving_requests() { + let temp = tempfile::tempdir().unwrap(); + let request = ObserveRequest::new("h.worker".into(), "queue".into(), None, None).unwrap(); + let path = request_path(temp.path(), &request.request_id).unwrap(); + write_json_atomically_no_fsync(&path, &request).unwrap(); + for index in 0..MAX_PENDING_OBSERVE_REQUESTS { + fs::write( + temp.path().join(format!(".observe-crash-{index}")), + b"incomplete", + ) + .unwrap(); + } + fs::write(temp.path().join("sibling"), b"not json").unwrap(); + + let (records, errors) = scan_requests(temp.path()); + + assert!(errors.is_empty()); + assert_eq!(records.len(), 1); + assert_eq!(records[0].request, request); + assert!( + fs::read_dir(temp.path()) + .unwrap() + .flatten() + .all(|entry| !entry.file_name().to_string_lossy().starts_with(".observe-")) + ); + assert!(!durable_request_capacity_is_full(temp.path()).unwrap()); + for invalid in ["../escape", "a/b", ".hidden", "x.json", "with space"] { + assert!( + request_path(temp.path(), invalid).is_err(), + "accepted {invalid:?}" + ); + } + } + + #[test] + fn receipt_taxonomy_is_terminal_only_on_evidence() { + assert!(!ObserveReceiptStatus::Accepted.is_terminal()); + assert!(!ObserveReceiptStatus::Backpressured.is_terminal()); + assert!(ObserveReceiptStatus::SettledUnchanged.is_terminal()); + assert!(ObserveReceiptStatus::SettledChanged.is_success()); + assert!(!ObserveReceiptStatus::SettledFailed.is_success()); + assert!(!ObserveReceiptStatus::ProviderUnavailable.is_success()); + } + + #[test] + fn receipt_status_keeps_human_and_wire_spellings_distinct() { + assert_eq!( + ObserveReceiptStatus::SettledUnchanged.as_str(), + "settled-unchanged" + ); + assert_eq!( + ObserveReceiptStatus::SettledUnchanged.wire_str(), + "settledUnchanged" + ); + assert_eq!( + ObserveReceiptStatus::ProviderUnavailable.wire_str(), + "providerUnavailable" + ); + } + #[test] + fn receipt_evidence_shape_matches_atomic_results() { + let request = + ObserveRequest::new("h.worker".into(), "queue".into(), Some(9), None).unwrap(); + let authority = ObservationAuthority { + owner: RuntimeOwner::new( + crate::resource_profile::RuntimeIncarnation::new("incarnation").unwrap(), + crate::resource_profile::OwnerClaim::new("claim").unwrap(), + ), + binding_id: BindingId::new("binding").unwrap(), + registration: RegistrationToken::new("registration").unwrap(), + }; + + let changed_without_digest = ObserveReceipt::new( + &request, + ObserveReceiptStatus::SettledChanged, + Some(authority.clone()), + Some(1), + None, + None, + ) + .unwrap(); + assert!(changed_without_digest.validate().is_err()); + + let digest = SnapshotDigest::of(b"accepted publication"); + let unchanged_with_digest = ObserveReceipt::new( + &request, + ObserveReceiptStatus::SettledUnchanged, + Some(authority.clone()), + Some(1), + Some(digest), + None, + ) + .unwrap(); + assert!(unchanged_with_digest.validate().is_err()); + let predispatch_stale = ObserveReceipt::new( + &request, + ObserveReceiptStatus::StaleGeneration, + None, + None, + None, + Some("catalog changed".into()), + ) + .unwrap(); + predispatch_stale.validate().unwrap(); + + let active_stale = ObserveReceipt::new( + &request, + ObserveReceiptStatus::StaleGeneration, + Some(authority.clone()), + Some(1), + None, + Some("registration changed".into()), + ) + .unwrap(); + active_stale.validate().unwrap(); + + let partial_authority = ObserveReceipt::new( + &request, + ObserveReceiptStatus::ProviderUnavailable, + Some(authority.clone()), + None, + None, + Some("runtime exited".into()), + ) + .unwrap(); + assert!(partial_authority.validate().is_err()); + + let changed = ObserveReceipt::new( + &request, + ObserveReceiptStatus::SettledChanged, + Some(authority), + Some(1), + Some(digest), + None, + ) + .unwrap(); + changed.validate().unwrap(); + } + + #[test] + fn receipt_diagnostics_are_normalized_to_safe_optional_text() { + let request = + ObserveRequest::new("h.worker".into(), "queue".into(), Some(9), None).unwrap(); + let empty = ObserveReceipt::new( + &request, + ObserveReceiptStatus::ProviderUnavailable, + None, + None, + None, + Some(String::new()), + ) + .unwrap(); + assert_eq!(empty.diagnostic, None); + + let with_nul = ObserveReceipt::new( + &request, + ObserveReceiptStatus::ProviderUnavailable, + None, + None, + None, + Some("provider\0refused".to_owned()), + ) + .unwrap(); + assert_eq!(with_nul.diagnostic.as_deref(), Some("provider�refused")); + with_nul.validate().unwrap(); + } + + #[test] + fn request_scan_is_bounded_to_the_durable_admission_limit() { + let temp = tempfile::tempdir().unwrap(); + for index in 0..(MAX_PENDING_OBSERVE_REQUESTS + 44) { + let mut request = + ObserveRequest::new("h.worker".into(), "queue".into(), Some(9), None).unwrap(); + request.request_id = format!("bounded-{index:03}"); + let path = request_path(temp.path(), &request.request_id).unwrap(); + write_json_atomically_no_fsync(&path, &request).unwrap(); + } + + let (records, errors) = scan_requests(temp.path()); + assert!(errors.is_empty(), "{errors:?}"); + assert_eq!(records.len(), MAX_PENDING_OBSERVE_REQUESTS); + } + + #[test] + fn unknown_control_fields_are_rejected() { + let raw = br#"{"schema":"st2.resource-observe-request.v1","requestId":"one","recipient":"h.a","binding":"x","requestedAt":"now","extra":true}"#; + assert!(serde_json::from_slice::(raw).is_err()); + } +} diff --git a/src/resource_profile.rs b/src/resource_profile.rs index 81fdf000..282554ae 100644 --- a/src/resource_profile.rs +++ b/src/resource_profile.rs @@ -17,11 +17,12 @@ use std::sync::atomic::{AtomicU64, Ordering}; use serde::{Deserialize, Serialize}; pub use st2_resource_protocol::{ - BindingId, FactError, FactValue, HostMessage, OpaqueIdError, OwnerClaim, ProtocolError, - RegistrationToken, ResourceFact, RuntimeHealthState, RuntimeIncarnation, RuntimeMessage, - RuntimeOwner, SnapshotBytes, SnapshotDigest, SnapshotSizeError, MAX_FACTS, MAX_FACT_KEY_BYTES, - MAX_FACT_VALUE_BYTES, MAX_HEALTH_DETAIL_BYTES, MAX_PROTOCOL_LINE_BYTES, MAX_SELECTOR_BYTES, - MAX_SNAPSHOT_BYTES, decode_host_line, decode_runtime_line, encode_host_line, + BindingId, FactError, FactValue, HostMessage, MAX_FACT_KEY_BYTES, MAX_FACT_VALUE_BYTES, + MAX_FACTS, MAX_HEALTH_DETAIL_BYTES, MAX_OBSERVATION_DIAGNOSTIC_BYTES, MAX_PROTOCOL_LINE_BYTES, + MAX_SELECTOR_BYTES, MAX_SNAPSHOT_BYTES, ObservationResult, OpaqueIdError, OwnerClaim, + ProtocolError, Publication, RegistrationToken, ResourceFact, RuntimeHealthState, + RuntimeIncarnation, RuntimeMessage, RuntimeOwner, SnapshotBytes, SnapshotDigest, + SnapshotSizeError, decode_host_line, decode_runtime_line, encode_host_line, encode_runtime_line, }; @@ -161,7 +162,10 @@ pub struct SnapshotTarget { } impl SnapshotTarget { - pub fn new(root: impl Into, carrier_path: impl AsRef) -> Result { + pub fn new( + root: impl Into, + carrier_path: impl AsRef, + ) -> Result { let root = root.into(); validate_absolute_path(&root).map_err(PathError::UnsafeRoot)?; let carrier_path = carrier_path.as_ref(); @@ -192,10 +196,9 @@ impl SnapshotTarget { /// Digest the currently published contained snapshot, if present. pub fn current_digest(&self) -> Result, PublicationError> { let parent = self.relative.parent().unwrap_or_else(|| Path::new("")); - let leaf = self - .relative - .file_name() - .ok_or_else(|| PublicationError::UnsafeTarget(PathError::UnsafeCarrier("missing leaf")))?; + let leaf = self.relative.file_name().ok_or_else(|| { + PublicationError::UnsafeTarget(PathError::UnsafeCarrier("missing leaf")) + })?; let directory = match open_absolute_dir_beneath(&self.root, parent) { Ok(directory) => directory, Err(error) if error.kind() == io::ErrorKind::NotFound => return Ok(None), @@ -366,36 +369,12 @@ impl RuntimeLifecycle { owner, binding_id, registration, - schema_id, - media_type, - bytes, - topics, - facts, - observed_at, + publication, } => { let binding = self.require_registration(owner, binding_id, registration)?; - if schema_id != binding.contract.schema_id() { - return Err(FenceError::ContractMismatch { field: "schemaId" }); - } - if media_type != binding.contract.media_type() { - return Err(FenceError::ContractMismatch { field: "mediaType" }); - } - let mut unique = BTreeSet::new(); - for topic in topics { - if topic.is_empty() || !unique.insert(topic.as_str()) { - return Err(FenceError::InvalidTopics); - } - if !binding.contract.published_topics.contains(topic) { - return Err(FenceError::UnpublishedTopic(topic.clone())); - } - } - Ok(AcceptedOutput::Publication(AcceptedPublication { - target: &binding.target, - bytes, - selected_topics: binding.contract.selection.select(topics), - facts: facts.as_deref().unwrap_or_default(), - observed_at: observed_at.as_deref(), - })) + Ok(AcceptedOutput::Publication( + self.accept_publication(binding, publication)?, + )) } RuntimeMessage::Health { owner, @@ -424,9 +403,73 @@ impl RuntimeLifecycle { detail: detail.as_deref(), })) } + RuntimeMessage::ObservationResult { + owner, + binding_id, + registration, + demand_watermark, + result, + } => { + let binding = self.require_registration(owner, binding_id, registration)?; + if *demand_watermark == 0 { + return Err(FenceError::InvalidDemandWatermark); + } + let result = match result { + ObservationResult::Unchanged => AcceptedObservation::Unchanged, + ObservationResult::Failed { diagnostic } => { + if diagnostic.as_ref().is_some_and(|diagnostic| { + diagnostic.len() > MAX_OBSERVATION_DIAGNOSTIC_BYTES + }) { + return Err(FenceError::ObservationDiagnosticTooLarge); + } + AcceptedObservation::Failed { + diagnostic: diagnostic.as_deref(), + } + } + ObservationResult::Published { publication } => AcceptedObservation::Published( + self.accept_publication(binding, publication)?, + ), + }; + Ok(AcceptedOutput::ObservationResult( + AcceptedObservationResult { + binding_id, + demand_watermark: *demand_watermark, + result, + }, + )) + } } } + fn accept_publication<'a>( + &'a self, + binding: &'a BindingRegistration, + publication: &'a Publication, + ) -> Result, FenceError> { + if publication.schema_id != binding.contract.schema_id() { + return Err(FenceError::ContractMismatch { field: "schemaId" }); + } + if publication.media_type != binding.contract.media_type() { + return Err(FenceError::ContractMismatch { field: "mediaType" }); + } + let mut unique = BTreeSet::new(); + for topic in &publication.topics { + if topic.is_empty() || !unique.insert(topic.as_str()) { + return Err(FenceError::InvalidTopics); + } + if !binding.contract.published_topics.contains(topic) { + return Err(FenceError::UnpublishedTopic(topic.clone())); + } + } + Ok(AcceptedPublication { + binding_id: &binding.binding_id, + target: &binding.target, + bytes: &publication.bytes, + selected_topics: binding.contract.selection.select(&publication.topics), + facts: publication.facts.as_deref().unwrap_or_default(), + }) + } + fn require_owner(&self, owner: &RuntimeOwner) -> Result<(), FenceError> { match self.owner.as_ref() { None => Err(FenceError::NoOwner), @@ -463,7 +506,9 @@ pub enum FenceError { UnpublishedTopic(String), InvalidTopics, InvalidHealthScope, + InvalidDemandWatermark, HealthDetailTooLarge, + ObservationDiagnosticTooLarge, } impl fmt::Display for FenceError { @@ -483,7 +528,13 @@ impl fmt::Display for FenceError { } Self::InvalidTopics => formatter.write_str("runtime output has invalid topics"), Self::InvalidHealthScope => formatter.write_str("runtime health has an invalid scope"), + Self::InvalidDemandWatermark => { + formatter.write_str("runtime observation result has an invalid demand watermark") + } Self::HealthDetailTooLarge => formatter.write_str("runtime health detail is too large"), + Self::ObservationDiagnosticTooLarge => { + formatter.write_str("runtime observation diagnostic is too large") + } } } } @@ -494,18 +545,23 @@ impl std::error::Error for FenceError {} pub enum AcceptedOutput<'a> { Publication(AcceptedPublication<'a>), Health(AcceptedHealth<'a>), + ObservationResult(AcceptedObservationResult<'a>), } #[derive(Debug)] pub struct AcceptedPublication<'a> { + binding_id: &'a BindingId, target: &'a SnapshotTarget, bytes: &'a SnapshotBytes, selected_topics: Vec, facts: &'a [ResourceFact], - observed_at: Option<&'a str>, } impl<'a> AcceptedPublication<'a> { + pub fn binding_id(&self) -> &BindingId { + self.binding_id + } + pub fn target(&self) -> &SnapshotTarget { self.target } @@ -517,9 +573,6 @@ impl<'a> AcceptedPublication<'a> { pub fn facts(&self) -> &[ResourceFact] { self.facts } - pub fn observed_at(&self) -> Option<&str> { - self.observed_at - } fn prepare(self) -> Result, PublicationError> { prepare_snapshot( @@ -531,6 +584,26 @@ impl<'a> AcceptedPublication<'a> { } } +#[derive(Debug)] +pub struct AcceptedObservationResult<'a> { + binding_id: &'a BindingId, + demand_watermark: u64, + result: AcceptedObservation<'a>, +} + +impl<'a> AcceptedObservationResult<'a> { + pub fn into_parts(self) -> (&'a BindingId, u64, AcceptedObservation<'a>) { + (self.binding_id, self.demand_watermark, self.result) + } +} + +#[derive(Debug)] +pub enum AcceptedObservation<'a> { + Unchanged, + Failed { diagnostic: Option<&'a str> }, + Published(AcceptedPublication<'a>), +} + #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub struct AcceptedHealth<'a> { binding_id: Option<&'a BindingId>, @@ -644,8 +717,8 @@ impl PreparedPublication<'_> { let leaf = self.target.relative.file_name().ok_or_else(|| { PublicationError::UnsafeTarget(PathError::UnsafeCarrier("missing leaf")) })?; - let directory = open_absolute_dir_beneath(&self.target.root, parent) - .map_err(PublicationError::Io)?; + let directory = + open_absolute_dir_beneath(&self.target.root, parent).map_err(PublicationError::Io)?; atomic_replace_at(&directory, leaf, self.bytes).map_err(PublicationError::Io) } } @@ -657,7 +730,9 @@ fn prepare_snapshot<'a>( facts: Vec, ) -> Result, PublicationError> { if bytes.len() > MAX_SNAPSHOT_BYTES { - return Err(PublicationError::SnapshotTooLarge { actual: bytes.len() }); + return Err(PublicationError::SnapshotTooLarge { + actual: bytes.len(), + }); } let parent = target.relative.parent().unwrap_or_else(|| Path::new("")); ensure_absolute_dir_beneath(&target.root, parent).map_err(PublicationError::Io)?; @@ -679,7 +754,7 @@ fn prepare_snapshot<'a>( }, }) } - +#[cfg(test)] fn publish_snapshot( target: &SnapshotTarget, bytes: &[u8], @@ -821,8 +896,7 @@ pub struct CatchUp { impl CatchUp { pub fn open(state_directory: &Path) -> Result { - validate_absolute_path(state_directory) - .map_err(CatchUpError::UnsafeStateDirectory)?; + validate_absolute_path(state_directory).map_err(CatchUpError::UnsafeStateDirectory)?; let directory = open_absolute_dir(state_directory).map_err(CatchUpError::Io)?; let state = match read_regular_optional_at( &directory, @@ -941,12 +1015,8 @@ impl CatchUp { }) } - pub fn acknowledge_delivery( - &mut self, - digest: SnapshotDigest, - ) -> Result { - if !self.state.pending_relevant_change - || self.state.current_snapshot_digest != Some(digest) + pub fn acknowledge_delivery(&mut self, digest: SnapshotDigest) -> Result { + if !self.state.pending_relevant_change || self.state.current_snapshot_digest != Some(digest) { return Ok(false); } @@ -993,22 +1063,15 @@ impl CatchUp { } } - fn write_publication_intent( - &self, - intent: &PublicationIntent, - ) -> Result<(), CatchUpError> { + fn write_publication_intent(&self, intent: &PublicationIntent) -> Result<(), CatchUpError> { intent.validate()?; let mut bytes = serde_json::to_vec(intent).map_err(CatchUpError::Json)?; bytes.push(b'\n'); if bytes.len() > MAX_CATCH_UP_FILE_BYTES { return Err(CatchUpError::IntentTooLarge); } - atomic_replace_at( - &self.directory, - OsStr::new(PUBLICATION_INTENT_FILE), - &bytes, - ) - .map_err(CatchUpError::Io) + atomic_replace_at(&self.directory, OsStr::new(PUBLICATION_INTENT_FILE), &bytes) + .map_err(CatchUpError::Io) } fn clear_publication_intent(&self) -> Result<(), CatchUpError> { @@ -1082,7 +1145,9 @@ impl fmt::Display for CatchUpError { formatter.write_str("publication intent path is not a real regular file") } Self::InvalidState(reason) => write!(formatter, "invalid catch-up state: {reason}"), - Self::Publication(error) => write!(formatter, "snapshot reconciliation failed: {error}"), + Self::Publication(error) => { + write!(formatter, "snapshot reconciliation failed: {error}") + } Self::Json(error) => write!(formatter, "invalid catch-up state JSON: {error}"), Self::Io(error) => write!(formatter, "catch-up state I/O failed: {error}"), } @@ -1142,10 +1207,13 @@ fn ensure_absolute_dir_beneath(root: &Path, relative: &Path) -> io::Result validate_absolute_path(root).map_err(invalid_input)?; validate_empty_or_relative_path(relative).map_err(invalid_input)?; let mut directory = open_absolute_dir(root)?; - for component in relative.components().filter_map(|component| match component { - Component::Normal(name) => Some(name), - _ => None, - }) { + for component in relative + .components() + .filter_map(|component| match component { + Component::Normal(name) => Some(name), + _ => None, + }) + { let name = c_string(component)?; let result = unsafe { libc::mkdirat(directory.as_raw_fd(), name.as_ptr(), 0o700) }; if result < 0 { @@ -1239,11 +1307,7 @@ fn atomic_replace_at(directory: &File, leaf: &OsStr, bytes: &[u8]) -> io::Result libc::openat( directory.as_raw_fd(), temporary.as_ptr(), - libc::O_WRONLY - | libc::O_CREAT - | libc::O_EXCL - | libc::O_NOFOLLOW - | libc::O_CLOEXEC, + libc::O_WRONLY | libc::O_CREAT | libc::O_EXCL | libc::O_NOFOLLOW | libc::O_CLOEXEC, 0o600, ) }; @@ -1287,7 +1351,6 @@ fn remove_optional_at(directory: &File, leaf: &OsStr) -> io::Result<()> { directory.sync_all() } - fn ensure_regular_or_absent_at(directory: &File, leaf: &OsStr) -> io::Result<()> { match read_regular_optional_at(directory, leaf, 0) { Ok(None) | Ok(Some(_)) | Err(BoundedReadError::TooLarge) => Ok(()), @@ -1362,12 +1425,17 @@ mod tests { owner, binding_id: binding_id("binding"), registration: token(registration), + publication: publication_payload(bytes, topics), + } + } + + fn publication_payload(bytes: &[u8], topics: &[&str]) -> Publication { + Publication { schema_id: "schema.v1".to_owned(), media_type: "application/json".to_owned(), bytes: SnapshotBytes::new(bytes.to_vec()).unwrap(), topics: topics.iter().map(|topic| (*topic).to_owned()).collect(), facts: None, - observed_at: None, } } @@ -1378,10 +1446,10 @@ mod tests { match lifecycle.accept_output(message).unwrap() { AcceptedOutput::Publication(publication) => publication, AcceptedOutput::Health(_) => panic!("expected publication"), + AcceptedOutput::ObservationResult(_) => panic!("expected publication"), } } - #[test] fn stale_owner_and_registration_are_fenced() { let directory = tempfile::tempdir().unwrap(); @@ -1398,11 +1466,35 @@ mod tests { lifecycle.accept_output(&stale_owner), Err(FenceError::StaleOwner) )); - let stale_token = publication(current, "stale-token", b"bytes", &["selected"]); + let stale_token = publication(current.clone(), "stale-token", b"bytes", &["selected"]); assert!(matches!( lifecycle.accept_output(&stale_token), Err(FenceError::StaleRegistration) )); + let stale_owner_result = RuntimeMessage::ObservationResult { + owner: owner("stale"), + binding_id: binding_id("binding"), + registration: token("current-token"), + demand_watermark: 1, + result: ObservationResult::Unchanged, + }; + assert!(matches!( + lifecycle.accept_output(&stale_owner_result), + Err(FenceError::StaleOwner) + )); + let stale_registration_result = RuntimeMessage::ObservationResult { + owner: current, + binding_id: binding_id("binding"), + registration: token("stale-token"), + demand_watermark: 1, + result: ObservationResult::Published { + publication: publication_payload(b"demand", &["selected"]), + }, + }; + assert!(matches!( + lifecycle.accept_output(&stale_registration_result), + Err(FenceError::StaleRegistration) + )); assert!(!directory.path().join("snapshot.json").exists()); } @@ -1475,7 +1567,10 @@ mod tests { .unwrap(); assert_eq!(first.change(), SnapshotChange::First); assert!(first.invalidating()); - assert_eq!(fs::read(directory.path().join("snapshot.json")).unwrap(), br#"{"state":1}"#); + assert_eq!( + fs::read(directory.path().join("snapshot.json")).unwrap(), + br#"{"state":1}"# + ); let (equal, _) = catch_up .publish(accepted_publication(&lifecycle, &message)) @@ -1521,12 +1616,7 @@ mod tests { assert!(outcome.selected_topics().is_empty()); assert!(!outcome.invalidating()); - let selected = publication( - current, - "token", - b"selected", - &["ignored", "selected"], - ); + let selected = publication(current, "token", b"selected", &["ignored", "selected"]); let (outcome, _) = catch_up .publish(accepted_publication(&lifecycle, &selected)) .unwrap(); @@ -1598,10 +1688,7 @@ mod tests { Some(outcome.digest()) ); assert!(catch_up.state().pending_relevant_change()); - assert_eq!( - catch_up.state().pending_selected_topics(), - ["selected"] - ); + assert_eq!(catch_up.state().pending_selected_topics(), ["selected"]); assert_eq!(catch_up.state().pending_facts(), outcome.facts()); } @@ -1657,13 +1744,9 @@ mod tests { } let snapshot_target = target(directory.path()); - let mut catch_up = - CatchUp::open_for_snapshot(&state_directory, &snapshot_target).unwrap(); + let mut catch_up = CatchUp::open_for_snapshot(&state_directory, &snapshot_target).unwrap(); assert!(catch_up.state().pending_relevant_change()); - assert_eq!( - catch_up.state().pending_selected_topics(), - ["selected"] - ); + assert_eq!(catch_up.state().pending_selected_topics(), ["selected"]); let (equal, _) = catch_up .publish(accepted_publication(&lifecycle, &message)) diff --git a/src/resource_profile_supervisor.rs b/src/resource_profile_supervisor.rs index 84beb250..d452cd4d 100644 --- a/src/resource_profile_supervisor.rs +++ b/src/resource_profile_supervisor.rs @@ -11,22 +11,30 @@ use std::process::{Child, Command, Stdio}; use std::sync::atomic::{AtomicU64, Ordering}; use std::sync::mpsc::{self, Receiver, SyncSender, TrySendError}; use std::thread::{self, JoinHandle}; +use std::time::{Duration, Instant, SystemTime}; use agent_spec::profile::{ ProfileCapability, ProfileDescriptor, ResourceProfileRegistry, RuntimeTopology, }; use agent_spec::spec::AgentSpec; use anyhow::Context as _; +use notify::RecommendedWatcher; use serde::Serialize; use serde_json::Value; use sha2::{Digest as _, Sha256}; use crate::catalog::CatalogConfig; +use crate::resource_observe::{ + ObservationAuthority, ObserveReceipt, ObserveReceiptStatus, ObserveRequest, + PendingRequestRecord, prepare_scope, prune_terminal_receipts, read_receipt, remove_request, + scan_requests, write_receipt, +}; use crate::resource_profile::{ - AcceptedOutput, BindingId, BindingRegistration, CatchUp, HostMessage, OwnerClaim, - PublicationContract, RegistrationToken, ResourceFact, RuntimeHealthState, RuntimeIncarnation, + AcceptedObservation, AcceptedOutput, AcceptedPublication, BindingId, BindingRegistration, + CatchUp, HostMessage, MAX_PROTOCOL_LINE_BYTES, OwnerClaim, PublicationContract, + PublicationOutcome, RegistrationToken, ResourceFact, RuntimeHealthState, RuntimeIncarnation, RuntimeLifecycle, RuntimeMessage, RuntimeOwner, SnapshotDigest, SnapshotTarget, TopicSelection, - MAX_PROTOCOL_LINE_BYTES, decode_runtime_line, encode_host_line, + decode_runtime_line, encode_host_line, }; const MAILBOX_CAPACITY: usize = 64; @@ -51,25 +59,66 @@ pub struct ResourceProfileSupervisor { worker: Option>, catalog_root: PathBuf, this_host: String, + observe_watcher: Option, + observe_bridge: Option>, } impl ResourceProfileSupervisor { pub fn new(catalog_root: PathBuf, this_host: String) -> anyhow::Result { let catalog_root = lexical_absolute(&catalog_root)?; + let scope = crate::park::SupervisorScope::current(&catalog_root, &this_host)?; + prepare_scope(&scope)?; + let request_dir = scope.observe_request_dir(); + let receipt_dir = scope.observe_receipt_dir(); + let (watch_tx, watch_rx) = mpsc::channel(); + let observe_watcher = crate::watch::watch_recursive_mutations(&request_dir, watch_tx); + if observe_watcher.is_none() { + tracing::warn!( + "Resource observation request watcher is unavailable; \ + falling back to supervisor refreshes" + ); + } let (tx, rx) = mpsc::sync_channel(MAILBOX_CAPACITY); let worker_tx = tx.clone(); let worker_root = catalog_root.clone(); let worker_host = this_host.clone(); let worker = thread::Builder::new() .name("st2-resource-profile".to_owned()) - .spawn(move || Worker::new(worker_root, worker_host, worker_tx).run(rx)) + .spawn(move || { + Worker::new( + worker_root, + worker_host, + worker_tx, + request_dir, + receipt_dir, + ) + .run(rx); + }) .context("spawn Resource Profile supervisor")?; - Ok(Self { + let bridge_tx = tx.clone(); + let observe_bridge = thread::Builder::new() + .name("st2-resource-observe-watch".to_owned()) + .spawn(move || { + while watch_rx.recv().is_ok() { + if bridge_tx.send(Msg::ObserveRequests).is_err() { + break; + } + } + }) + .context("spawn Resource observation request watcher")?; + let supervisor = Self { tx, worker: Some(worker), catalog_root, this_host, - }) + observe_watcher, + observe_bridge: Some(observe_bridge), + }; + // Watch-before-scan: when the watcher is available, a final rename racing startup is + // either in this scan or queued by the watcher. Without a watcher, later supervisor + // refreshes continue to scan the durable request directory. + let _ = supervisor.tx.send(Msg::ObserveRequests); + Ok(supervisor) } /// Reconcile the exact set of bindings owned by canonical agent seats proven live this pass. @@ -94,6 +143,7 @@ impl ResourceProfileSupervisor { .tx .send(Msg::Refresh { desired, + catalog_generation: generation, reply: reply_tx, }) .is_err() @@ -134,6 +184,7 @@ impl ResourceProfileSupervisor { impl Drop for ResourceProfileSupervisor { fn drop(&mut self) { + self.observe_watcher.take(); let (reply_tx, reply_rx) = mpsc::sync_channel(1); if self.tx.send(Msg::Shutdown { reply: reply_tx }).is_ok() { let _ = reply_rx.recv(); @@ -141,6 +192,9 @@ impl Drop for ResourceProfileSupervisor { if let Some(worker) = self.worker.take() { let _ = worker.join(); } + if let Some(bridge) = self.observe_bridge.take() { + let _ = bridge.join(); + } } } @@ -148,6 +202,7 @@ impl Drop for ResourceProfileSupervisor { enum Msg { Refresh { desired: BTreeMap, + catalog_generation: Option, reply: SyncSender>, }, Deactivate { @@ -166,11 +221,16 @@ enum Msg { key: RuntimeKey, owner: RuntimeOwner, }, + WriterDrained { + key: RuntimeKey, + owner: RuntimeOwner, + }, WriterFailed { key: RuntimeKey, owner: RuntimeOwner, error: String, }, + ObserveRequests, Shutdown { reply: SyncSender<()>, }, @@ -187,6 +247,7 @@ struct DesiredBinding { generation: u64, topology: RuntimeTopology, argv: Vec, + demand: bool, uri: String, selector: Value, descriptor: ProfileDescriptor, @@ -242,7 +303,9 @@ fn desired_bindings( }; match refresh.try_descriptor(&declared.scheme) { Ok(Some(descriptor)) - if descriptor.capabilities.contains(&ProfileCapability::Observe) => + if descriptor + .capabilities + .contains(&ProfileCapability::Observe) => { generations.insert( declared.scheme.clone(), @@ -314,10 +377,7 @@ fn desired_bindings( continue; } }; - let target = match SnapshotTarget::new( - resolution.containment_root, - &resolution.path, - ) { + let target = match SnapshotTarget::new(resolution.containment_root, &resolution.path) { Ok(target) => target, Err(error) => { warnings.push(format!( @@ -343,6 +403,7 @@ fn desired_bindings( generation: *generation, topology: descriptor.runtime.topology, argv: runtime.argv.clone(), + demand: runtime.demand, uri: resource.uri().to_owned(), selector, descriptor: descriptor.clone(), @@ -358,28 +419,56 @@ struct Worker { catalog_root: PathBuf, this_host: String, tx: SyncSender, + request_dir: PathBuf, + receipt_dir: PathBuf, + initialized: bool, + catalog_generation: Option, + desired: BTreeMap, runtimes: BTreeMap, } impl Worker { - fn new(catalog_root: PathBuf, this_host: String, tx: SyncSender) -> Self { + fn new( + catalog_root: PathBuf, + this_host: String, + tx: SyncSender, + request_dir: PathBuf, + receipt_dir: PathBuf, + ) -> Self { Self { catalog_root, this_host, tx, + request_dir, + receipt_dir, + initialized: false, + catalog_generation: None, + desired: BTreeMap::new(), runtimes: BTreeMap::new(), } } fn run(mut self, rx: Receiver) { while let Ok(message) = rx.recv() { + let mut scan_requests_now = false; match message { - Msg::Refresh { desired, reply } => { + Msg::Refresh { + desired, + catalog_generation, + reply, + } => { + self.catalog_generation = catalog_generation; + self.desired = desired.clone(); let warnings = self.reconcile(desired); + self.initialized = true; + scan_requests_now = true; let _ = reply.send(warnings); } Msg::Deactivate { recipient, reply } => { self.deactivate_recipient(&recipient); + self.desired + .retain(|_, binding| binding.recipient != recipient); + scan_requests_now = true; let _ = reply.send(()); } Msg::Health { reply } => { @@ -392,12 +481,26 @@ impl Worker { } Msg::RuntimeOutput { key, owner, output } => { self.runtime_output(&key, &owner, output); + scan_requests_now = true; } Msg::RuntimeEof { key, owner } => { self.runtime_failed(&key, &owner, "runtime protocol reached EOF"); + scan_requests_now = true; + } + Msg::WriterDrained { key, owner } => { + if !owner_matches( + self.runtimes.get(&key).map(|runtime| &runtime.owner), + &owner, + ) { + continue; + } } Msg::WriterFailed { key, owner, error } => { self.runtime_failed(&key, &owner, &error); + scan_requests_now = true; + } + Msg::ObserveRequests => { + scan_requests_now = true; } Msg::Shutdown { reply } => { self.stop_all(); @@ -405,6 +508,11 @@ impl Worker { break; } } + self.retry_observe_dispatches(); + if scan_requests_now { + self.consume_observe_requests(); + self.retry_observe_dispatches(); + } } self.stop_all(); } @@ -423,6 +531,10 @@ impl Worker { .collect::>(); for key in obsolete { if let Some(mut runtime) = self.runtimes.remove(&key) { + runtime.finalize_all_demand( + ObserveReceiptStatus::StaleGeneration, + Some("binding generation was replaced".to_owned()), + ); runtime.deactivate_all(); runtime.stop(); } @@ -430,7 +542,10 @@ impl Worker { let mut grouped: BTreeMap> = BTreeMap::new(); for binding in desired.into_values() { - grouped.entry(binding.runtime_key()).or_default().push(binding); + grouped + .entry(binding.runtime_key()) + .or_default() + .push(binding); } for (key, bindings) in grouped { if !self.runtimes.contains_key(&key) { @@ -462,6 +577,10 @@ impl Worker { runtime.scheme )); if let Some(mut failed) = self.runtimes.remove(&key) { + failed.finalize_all_demand( + ObserveReceiptStatus::ProviderUnavailable, + Some("runtime registration failed".to_owned()), + ); failed.stop(); } } @@ -484,6 +603,172 @@ impl Worker { } } } + fn retry_observe_dispatches(&mut self) { + for runtime in self.runtimes.values_mut() { + for error in runtime.retry_observe_dispatches() { + eprintln!("st2: Resource Profile '{}': {error}", runtime.scheme); + } + } + } + + fn consume_observe_requests(&mut self) { + if !self.initialized { + return; + } + let _span = tracing::info_span!( + "resource.observe.consume", + catalog = %self.catalog_root.display(), + host = %self.this_host + ) + .entered(); + let (records, errors) = scan_requests(&self.request_dir); + for error in errors { + eprintln!("st2: {error}"); + } + for record in records { + match read_receipt(&self.receipt_dir, &record.request.request_id) { + Ok(Some(receipt)) if receipt.status.is_terminal() => { + let _ = remove_request(&record.path); + continue; + } + Ok(_) => {} + Err(error) => { + eprintln!( + "st2: reading observe receipt for {:?}: {error:#}", + record.request.request_id + ); + continue; + } + } + if self + .runtimes + .values() + .any(|runtime| runtime.contains_request(&record.request.request_id)) + { + continue; + } + if let Some(expected) = record.request.expected_catalog_generation { + match self.catalog_generation { + Some(current) if expected < current => { + self.finish_request_without_dispatch( + &record, + ObserveReceiptStatus::StaleGeneration, + "catalog generation changed before dispatch", + ); + continue; + } + Some(current) if expected > current => continue, + None => continue, + Some(_) => {} + } + } + let stable_key = record.request.stable_key(); + let runtime_key = self.runtimes.iter().find_map(|(key, runtime)| { + runtime + .bindings + .contains_key(&stable_key) + .then(|| key.clone()) + }); + let Some(runtime_key) = runtime_key else { + let (status, detail) = match self.desired.get(&stable_key) { + Some(desired) if desired.demand => ( + ObserveReceiptStatus::ProviderUnavailable, + "the declared provider runtime is not available", + ), + Some(_) => ( + ObserveReceiptStatus::AbsentBinding, + "the profile runtime does not declare the demand capability", + ), + None => ( + ObserveReceiptStatus::AbsentBinding, + "no active observable binding matches the target", + ), + }; + self.finish_request_without_dispatch(&record, status, detail); + continue; + }; + let Some(active) = self + .runtimes + .get(&runtime_key) + .and_then(|runtime| runtime.bindings.get(&stable_key)) + else { + continue; + }; + if !active.desired.demand { + self.finish_request_without_dispatch( + &record, + ObserveReceiptStatus::AbsentBinding, + "the profile runtime does not declare the demand capability", + ); + continue; + } + if record.request.expected_snapshot_digest.is_some() + && record.request.expected_snapshot_digest + != active.catch_up.state().current_snapshot_digest() + { + self.finish_request_without_dispatch( + &record, + ObserveReceiptStatus::StaleGeneration, + "snapshot digest changed before dispatch", + ); + continue; + } + let enqueue = self + .runtimes + .get_mut(&runtime_key) + .context("resolved runtime disappeared before demand enqueue") + .and_then(|runtime| { + runtime.enqueue_demand( + stable_key, + record.request.clone(), + record.path.clone(), + record.modified_at, + ) + }); + if let Err(error) = enqueue { + self.finish_request_without_dispatch( + &record, + ObserveReceiptStatus::ProviderUnavailable, + &error.to_string(), + ); + continue; + } + } + for error in prune_terminal_receipts(&self.receipt_dir) { + eprintln!("st2: {error}"); + } + } + + fn finish_request_without_dispatch( + &self, + record: &PendingRequestRecord, + status: ObserveReceiptStatus, + diagnostic: &str, + ) { + let receipt = ObserveReceipt::new( + &record.request, + status, + None, + None, + None, + Some(diagnostic.to_owned()), + ); + match receipt.and_then(|receipt| write_receipt(&self.receipt_dir, &receipt)) { + Ok(()) => { + crate::metrics::record_resource_observe_request(status.wire_str()); + if let Err(error) = remove_request(&record.path) { + eprintln!( + "st2: consuming observe request {}: {error:#}", + record.path.display() + ); + } + } + Err(error) => eprintln!( + "st2: writing observe receipt for {:?}: {error:#}", + record.request.request_id + ), + } + } fn runtime_output( &mut self, @@ -515,13 +800,16 @@ impl Worker { } if let Some(mut runtime) = self.runtimes.remove(key) { eprintln!("st2: Resource Profile '{}': {detail}", runtime.scheme); + runtime.finalize_all_demand( + ObserveReceiptStatus::ProviderUnavailable, + Some(detail.to_owned()), + ); runtime.stop(); } } fn stop_all(&mut self) { for (_, mut runtime) in std::mem::take(&mut self.runtimes) { - runtime.deactivate_all(); runtime.stop(); } } @@ -536,6 +824,8 @@ struct RuntimeProcess { writer_thread: Option>, reader_thread: Option>, bindings: BTreeMap, + request_dir: PathBuf, + receipt_dir: PathBuf, process_health: ResourceProfileHealth, } @@ -545,6 +835,80 @@ struct ActiveBinding { registration: RegistrationToken, catch_up: CatchUp, health: ResourceProfileHealth, + demand: DemandState, +} + +#[derive(Debug, Default)] +struct DemandState { + next_watermark: u64, + in_flight: Option, + trailing: Option, + settled: Vec, +} + +#[derive(Debug)] +struct DemandBatch { + watermark: u64, + requests: Vec, + dispatched_at: Option, +} + +#[derive(Debug)] +struct DemandSettlement { + batch: DemandBatch, + status: ObserveReceiptStatus, + digest: Option, + diagnostic: Option, +} + +#[derive(Debug)] +struct PendingDemand { + request: ObserveRequest, + request_path: PathBuf, + queued_at: SystemTime, + last_status: Option, +} + +impl DemandState { + fn push( + &mut self, + request: ObserveRequest, + request_path: PathBuf, + queued_at: SystemTime, + ) -> anyhow::Result<()> { + if self.trailing.is_none() { + let watermark = self + .next_watermark + .checked_add(1) + .context("demand watermark exhausted")?; + self.next_watermark = watermark; + self.trailing = Some(DemandBatch { + watermark, + requests: Vec::new(), + dispatched_at: None, + }); + } + self.trailing + .as_mut() + .context("trailing demand batch is absent after allocation")? + .requests + .push(PendingDemand { + request, + request_path, + queued_at, + last_status: None, + }); + Ok(()) + } + + fn contains_request(&self, request_id: &str) -> bool { + self.in_flight + .iter() + .chain(self.trailing.iter()) + .chain(self.settled.iter().map(|settlement| &settlement.batch)) + .flat_map(|batch| &batch.requests) + .any(|pending| pending.request.request_id == request_id) + } } impl RuntimeProcess { @@ -565,7 +929,9 @@ impl RuntimeProcess { .stdin(Stdio::piped()) .stdout(Stdio::piped()) .stderr(Stdio::inherit()); - let mut child = command.spawn().with_context(|| format!("spawn {executable:?}"))?; + let mut child = command + .spawn() + .with_context(|| format!("spawn {executable:?}"))?; let stdin = child.stdin.take().context("capture runtime stdin")?; let stdout = child.stdout.take().context("capture runtime stdout")?; let sequence = ID_SEQUENCE.fetch_add(1, Ordering::Relaxed); @@ -587,13 +953,22 @@ impl RuntimeProcess { let writer_thread = thread::Builder::new() .name("st2-resource-profile-stdin".to_owned()) .spawn(move || { - runtime_writer(stdin, writer_rx, writer_key, writer_owner, writer_supervisor) + runtime_writer( + stdin, + writer_rx, + writer_key, + writer_owner, + writer_supervisor, + ) })?; let reader_key = key.clone(); let reader_owner = owner.clone(); let reader_thread = thread::Builder::new() .name("st2-resource-profile-stdout".to_owned()) .spawn(move || runtime_reader(stdout, reader_key, reader_owner, supervisor_tx))?; + let scope = crate::park::SupervisorScope::current(catalog_root, this_host)?; + let request_dir = scope.observe_request_dir(); + let receipt_dir = scope.observe_receipt_dir(); Ok(Self { scheme: sample.scheme.clone(), @@ -604,6 +979,8 @@ impl RuntimeProcess { writer_thread: Some(writer_thread), reader_thread: Some(reader_thread), bindings: BTreeMap::new(), + request_dir, + receipt_dir, process_health: ResourceProfileHealth { scheme: sample.scheme.clone(), binding: None, @@ -652,7 +1029,8 @@ impl RuntimeProcess { )))?; let registration = RegistrationToken::new(hash_text(&format!( "{}\0{}\0{sequence}", - self.owner.claim().as_str(), desired.stable_key + self.owner.claim().as_str(), + desired.stable_key )))?; let selection = selector_topics(&desired.selector)?; let contract = PublicationContract::new( @@ -700,6 +1078,7 @@ impl RuntimeProcess { binding_id, registration, catch_up, + demand: DemandState::default(), }; if pending.is_some() { let _ = emit_pending_for(&mut active); @@ -713,6 +1092,14 @@ impl RuntimeProcess { let Some(mut active) = self.bindings.remove(stable_key) else { return; }; + finalize_active_demand( + &self.request_dir, + &self.receipt_dir, + &self.owner, + &mut active, + ObserveReceiptStatus::StaleGeneration, + Some("binding registration was removed or replaced".to_owned()), + ); let _ = active.catch_up.set_deliverable(false); let message = HostMessage::Unregister { owner: self.owner.clone(), @@ -720,11 +1107,9 @@ impl RuntimeProcess { registration: active.registration.clone(), }; let _ = self.send(message); - let _ = self.lifecycle.unregister( - &self.owner, - &active.binding_id, - &active.registration, - ); + let _ = self + .lifecycle + .unregister(&self.owner, &active.binding_id, &active.registration); } fn deactivate_recipient(&mut self, recipient: &str) { @@ -747,26 +1132,150 @@ impl RuntimeProcess { } } + fn enqueue_demand( + &mut self, + stable_key: String, + request: ObserveRequest, + request_path: PathBuf, + queued_at: SystemTime, + ) -> anyhow::Result<()> { + self.bindings + .get_mut(&stable_key) + .context("active binding disappeared before demand enqueue")? + .demand + .push(request, request_path, queued_at) + } + + fn contains_request(&self, request_id: &str) -> bool { + self.bindings + .values() + .any(|active| active.demand.contains_request(request_id)) + } + + fn retry_observe_dispatches(&mut self) -> Vec { + let mut errors = Vec::new(); + for active in self.bindings.values_mut() { + let authority = ObservationAuthority { + owner: self.owner.clone(), + binding_id: active.binding_id.clone(), + registration: active.registration.clone(), + }; + errors.extend(retry_settled_demand( + &self.request_dir, + &self.receipt_dir, + &authority, + active, + )); + } + let keys = self + .bindings + .iter() + .filter(|(_, active)| { + active.demand.in_flight.is_none() && active.demand.trailing.is_some() + }) + .map(|(key, _)| key.clone()) + .collect::>(); + for key in keys { + let Some(active) = self.bindings.get(&key) else { + continue; + }; + let authority = ObservationAuthority { + owner: self.owner.clone(), + binding_id: active.binding_id.clone(), + registration: active.registration.clone(), + }; + let demand_watermark = active + .demand + .trailing + .as_ref() + .expect("binding was selected with trailing demand") + .watermark; + let message = HostMessage::Observe { + owner: authority.owner.clone(), + binding_id: authority.binding_id.clone(), + registration: authority.registration.clone(), + demand_watermark, + }; + match self.send(message) { + Ok(()) => { + let active = self + .bindings + .get_mut(&key) + .expect("binding selected from the same map"); + let mut batch = active + .demand + .trailing + .take() + .expect("binding was selected with trailing demand"); + batch.dispatched_at = Some(Instant::now()); + errors.extend(write_batch_status( + &self.request_dir, + &self.receipt_dir, + &authority, + &mut batch, + ObserveReceiptStatus::Accepted, + None, + None, + )); + for pending in &batch.requests { + crate::metrics::record_resource_observe_dispatch( + SystemTime::now() + .duration_since(pending.queued_at) + .unwrap_or(Duration::ZERO), + ); + } + active.demand.in_flight = Some(batch); + } + Err(error) => { + let active = self + .bindings + .get_mut(&key) + .expect("binding selected from the same map"); + let batch = active + .demand + .trailing + .as_mut() + .expect("binding was selected with trailing demand"); + errors.extend(write_batch_status( + &self.request_dir, + &self.receipt_dir, + &authority, + batch, + ObserveReceiptStatus::Backpressured, + None, + Some(error.to_string()), + )); + } + } + } + errors + } + + fn finalize_all_demand(&mut self, status: ObserveReceiptStatus, diagnostic: Option) { + for active in self.bindings.values_mut() { + finalize_active_demand( + &self.request_dir, + &self.receipt_dir, + &self.owner, + active, + status, + diagnostic.clone(), + ); + } + } + fn accept( &mut self, message: RuntimeMessage, catalog_root: &Path, this_host: &str, ) -> anyhow::Result<()> { - let binding_id = match &message { - RuntimeMessage::Publish { binding_id, .. } => Some(binding_id.clone()), - RuntimeMessage::Health { binding_id, .. } => binding_id.clone(), - }; match self.lifecycle.accept_output(&message)? { AcceptedOutput::Publication(publication) => { - let active = self - .bindings - .values_mut() - .find(|active| Some(&active.binding_id) == binding_id.as_ref()) - .context("accepted publication has no active binding")?; - let (_, pending) = active.catch_up.publish(publication)?; - if pending.is_some() { - emit_pending(catalog_root, this_host, active)?; + let (_, delivery_error) = + process_publication(&mut self.bindings, publication, catalog_root, this_host)?; + if let Some(error) = delivery_error { + return Err(error); } } AcceptedOutput::Health(health) => { @@ -784,6 +1293,63 @@ impl RuntimeProcess { self.process_health.detail = health.detail().map(str::to_owned); } } + AcceptedOutput::ObservationResult(observation) => { + let (binding_id, watermark, result) = observation.into_parts(); + { + let active = self + .bindings + .values() + .find(|active| &active.binding_id == binding_id) + .context("accepted observation result has no active binding")?; + validate_active_demand(active, watermark)?; + } + let (status, digest, diagnostic, delivery_error) = match result { + AcceptedObservation::Unchanged => { + (ObserveReceiptStatus::SettledUnchanged, None, None, None) + } + AcceptedObservation::Failed { diagnostic } => ( + ObserveReceiptStatus::SettledFailed, + None, + diagnostic.map(str::to_owned), + None, + ), + AcceptedObservation::Published(publication) => { + let (outcome, delivery_error) = process_publication( + &mut self.bindings, + publication, + catalog_root, + this_host, + )?; + ( + ObserveReceiptStatus::SettledChanged, + Some(outcome.digest()), + None, + delivery_error, + ) + } + }; + let active = self + .bindings + .values_mut() + .find(|active| &active.binding_id == binding_id) + .context("accepted observation result has no active binding")?; + settle_active_demand( + &self.request_dir, + &self.receipt_dir, + &self.owner, + active, + watermark, + status, + digest, + diagnostic, + )?; + if let Some(error) = delivery_error { + eprintln!( + "st2: Resource Profile '{}': demanded publication delivery failed: {error:#}", + self.scheme + ); + } + } } Ok(()) } @@ -817,6 +1383,195 @@ impl RuntimeProcess { } } +fn process_publication( + bindings: &mut BTreeMap, + publication: AcceptedPublication<'_>, + catalog_root: &Path, + this_host: &str, +) -> anyhow::Result<(PublicationOutcome, Option)> { + let binding_id = publication.binding_id().clone(); + let active = bindings + .values_mut() + .find(|active| active.binding_id == binding_id) + .context("accepted publication has no active binding")?; + let (outcome, pending) = active.catch_up.publish(publication)?; + let delivery_error = if pending.is_some() { + emit_pending(catalog_root, this_host, active).err() + } else { + None + }; + Ok((outcome, delivery_error)) +} + +fn validate_active_demand(active: &ActiveBinding, watermark: u64) -> anyhow::Result<()> { + let expected_watermark = active + .demand + .in_flight + .as_ref() + .context("runtime returned an observation result without an outstanding demand")? + .watermark; + anyhow::ensure!( + expected_watermark == watermark, + "runtime returned demand watermark {watermark}, expected {expected_watermark}" + ); + Ok(()) +} + +fn write_batch_status( + request_dir: &Path, + receipt_dir: &Path, + authority: &ObservationAuthority, + batch: &mut DemandBatch, + status: ObserveReceiptStatus, + digest: Option, + diagnostic: Option, +) -> Vec { + let mut errors = Vec::new(); + for pending in &mut batch.requests { + if pending.last_status == Some(status) { + continue; + } + let receipt = ObserveReceipt::new( + &pending.request, + status, + Some(authority.clone()), + Some(batch.watermark), + digest, + diagnostic.clone(), + ); + match receipt.and_then(|receipt| write_receipt(receipt_dir, &receipt)) { + Ok(()) => { + pending.last_status = Some(status); + crate::metrics::record_resource_observe_request(status.wire_str()); + if status.is_terminal() + && let Err(error) = remove_request(&pending.request_path) + { + errors.push(format!( + "removing terminal observe request {} from {}: {error:#}", + pending.request.request_id, + request_dir.display() + )); + } + } + Err(error) => errors.push(format!( + "writing observe receipt for {:?}: {error:#}", + pending.request.request_id + )), + } + } + errors +} + +fn retry_settled_demand( + request_dir: &Path, + receipt_dir: &Path, + authority: &ObservationAuthority, + active: &mut ActiveBinding, +) -> Vec { + let mut errors = Vec::new(); + for settlement in &mut active.demand.settled { + errors.extend(write_batch_status( + request_dir, + receipt_dir, + authority, + &mut settlement.batch, + settlement.status, + settlement.digest, + settlement.diagnostic.clone(), + )); + } + active.demand.settled.retain(|settlement| { + settlement + .batch + .requests + .iter() + .any(|pending| pending.last_status != Some(settlement.status)) + }); + errors +} + +fn settle_active_demand( + request_dir: &Path, + receipt_dir: &Path, + owner: &RuntimeOwner, + active: &mut ActiveBinding, + watermark: u64, + status: ObserveReceiptStatus, + digest: Option, + diagnostic: Option, +) -> anyhow::Result<()> { + validate_active_demand(active, watermark)?; + let batch = active + .demand + .in_flight + .take() + .context("validated outstanding demand disappeared before settlement")?; + if let Some(dispatched_at) = batch.dispatched_at { + for _ in &batch.requests { + crate::metrics::record_resource_observe_settle(dispatched_at.elapsed()); + } + } + let authority = ObservationAuthority { + owner: owner.clone(), + binding_id: active.binding_id.clone(), + registration: active.registration.clone(), + }; + active.demand.settled.push(DemandSettlement { + batch, + status, + digest, + diagnostic, + }); + for error in retry_settled_demand(request_dir, receipt_dir, &authority, active) { + eprintln!("st2: Resource Profile '{}': {error}", active.desired.scheme); + } + tracing::info!( + recipient = %active.desired.recipient, + binding = %active.desired.binding_name, + demand_watermark = watermark, + result = status.as_str(), + "Resource observation settled" + ); + Ok(()) +} + +fn finalize_active_demand( + request_dir: &Path, + receipt_dir: &Path, + owner: &RuntimeOwner, + active: &mut ActiveBinding, + status: ObserveReceiptStatus, + diagnostic: Option, +) { + let authority = ObservationAuthority { + owner: owner.clone(), + binding_id: active.binding_id.clone(), + registration: active.registration.clone(), + }; + for error in retry_settled_demand(request_dir, receipt_dir, &authority, active) { + eprintln!("st2: Resource Profile '{}': {error}", active.desired.scheme); + } + for mut batch in active + .demand + .in_flight + .take() + .into_iter() + .chain(active.demand.trailing.take()) + { + for error in write_batch_status( + request_dir, + receipt_dir, + &authority, + &mut batch, + status, + None, + diagnostic.clone(), + ) { + eprintln!("st2: Resource Profile '{}': {error}", active.desired.scheme); + } + } +} + fn runtime_writer( mut stdin: std::process::ChildStdin, rx: Receiver>, @@ -833,6 +1588,13 @@ fn runtime_writer( }); return; } + match supervisor.try_send(Msg::WriterDrained { + key: key.clone(), + owner: owner.clone(), + }) { + Ok(()) | Err(TrySendError::Full(_)) => {} + Err(TrySendError::Disconnected(_)) => return, + } } } @@ -879,7 +1641,11 @@ fn read_bounded_line(reader: &mut impl BufRead) -> io::Result>> { loop { let available = reader.fill_buf()?; if available.is_empty() { - return if line.is_empty() { Ok(None) } else { Ok(Some(line)) }; + return if line.is_empty() { + Ok(None) + } else { + Ok(Some(line)) + }; } if let Some(newline) = available.iter().position(|byte| *byte == b'\n') { let consumed = newline + 1; @@ -951,14 +1717,22 @@ fn profile_generation( declared.wasm, declared.class, declared.notify_chain, - declared.runtime.as_ref().map(|runtime| &runtime.argv), + declared.runtime.as_ref(), descriptor, ); let digest = Sha256::digest(input.as_bytes()); - u64::from_be_bytes(digest[..8].try_into().expect("SHA-256 prefix is eight bytes")) + u64::from_be_bytes( + digest[..8] + .try_into() + .expect("SHA-256 prefix is eight bytes"), + ) } -fn emit_pending(catalog_root: &Path, this_host: &str, active: &mut ActiveBinding) -> anyhow::Result<()> { +fn emit_pending( + catalog_root: &Path, + this_host: &str, + active: &mut ActiveBinding, +) -> anyhow::Result<()> { emit_pending_at(catalog_root, this_host, active) } @@ -1019,7 +1793,9 @@ fn emit_pending_at( } fn publication_event_id(recipient: &str, binding: &str, digest: SnapshotDigest) -> String { - hash_text(&format!("resource-profile\0{recipient}\0{binding}\0{digest}")) + hash_text(&format!( + "resource-profile\0{recipient}\0{binding}\0{digest}" + )) } pub(crate) fn resource_change_subject( @@ -1061,8 +1837,6 @@ pub(crate) fn resource_change_subject( return subject; } - // Facts and topic names are never clipped. A pathological oversized binding or topic suffix - // falls back to the binding and bounded generic detail; the durable body remains complete. let fallback = format!("{binding} · {fallback}"); if fallback.chars().count() <= SUBJECT_MAX_SCALARS { fallback @@ -1136,13 +1910,17 @@ fn hash_text(value: &str) -> String { #[cfg(test)] mod tests { use super::*; + use std::fs; #[test] fn bounded_reader_accepts_one_maximal_line_and_rejects_overflow() { let mut maximal = vec![b'x'; MAX_PROTOCOL_LINE_BYTES - 1]; maximal.push(b'\n'); assert_eq!( - read_bounded_line(&mut maximal.as_slice()).unwrap().unwrap().len(), + read_bounded_line(&mut maximal.as_slice()) + .unwrap() + .unwrap() + .len(), MAX_PROTOCOL_LINE_BYTES ); let mut overflow = vec![b'x'; MAX_PROTOCOL_LINE_BYTES]; @@ -1188,6 +1966,165 @@ mod tests { ); } + #[test] + fn demand_batches_coalesce_bursts_and_keep_one_trailing_watermark() { + let mut demand = DemandState::default(); + let first = ObserveRequest::new("h.a".into(), "one".into(), None, None).unwrap(); + let second = ObserveRequest::new("h.a".into(), "one".into(), None, None).unwrap(); + demand + .push(first, PathBuf::from("first.json"), SystemTime::now()) + .unwrap(); + demand + .push(second, PathBuf::from("second.json"), SystemTime::now()) + .unwrap(); + assert_eq!(demand.trailing.as_ref().unwrap().watermark, 1); + assert_eq!(demand.trailing.as_ref().unwrap().requests.len(), 2); + + let mut in_flight = demand.trailing.take().unwrap(); + in_flight.dispatched_at = Some(Instant::now()); + demand.in_flight = Some(in_flight); + demand + .push( + ObserveRequest::new("h.a".into(), "one".into(), None, None).unwrap(), + PathBuf::from("third.json"), + SystemTime::now(), + ) + .unwrap(); + demand + .push( + ObserveRequest::new("h.a".into(), "one".into(), None, None).unwrap(), + PathBuf::from("fourth.json"), + SystemTime::now(), + ) + .unwrap(); + assert_eq!(demand.in_flight.as_ref().unwrap().watermark, 1); + assert_eq!(demand.trailing.as_ref().unwrap().watermark, 2); + assert_eq!(demand.trailing.as_ref().unwrap().requests.len(), 2); + } + + #[test] + fn terminal_receipt_write_failure_preserves_the_batch_for_retry() { + let temporary = tempfile::tempdir().unwrap(); + let request_dir = temporary.path().join("requests"); + let receipt_dir = temporary.path().join("receipts"); + fs::create_dir_all(&request_dir).unwrap(); + fs::create_dir_all(&receipt_dir).unwrap(); + let request = ObserveRequest::new("h.a".into(), "one".into(), None, None).unwrap(); + let request_path = request_dir.join(format!("{}.json", request.request_id)); + fs::write(&request_path, serde_json::to_vec(&request).unwrap()).unwrap(); + let receipt_path = receipt_dir.join(format!("{}.json", request.request_id)); + fs::create_dir(&receipt_path).unwrap(); + let mut batch = DemandBatch { + watermark: 1, + requests: vec![PendingDemand { + request: request.clone(), + request_path: request_path.clone(), + queued_at: SystemTime::now(), + last_status: Some(ObserveReceiptStatus::Accepted), + }], + dispatched_at: Some(Instant::now()), + }; + let authority = ObservationAuthority { + owner: RuntimeOwner::new( + RuntimeIncarnation::new("incarnation").unwrap(), + OwnerClaim::new("claim").unwrap(), + ), + binding_id: BindingId::new("binding").unwrap(), + registration: RegistrationToken::new("registration").unwrap(), + }; + + let errors = write_batch_status( + &request_dir, + &receipt_dir, + &authority, + &mut batch, + ObserveReceiptStatus::SettledUnchanged, + None, + None, + ); + assert_eq!(errors.len(), 1); + assert_eq!( + batch.requests[0].last_status, + Some(ObserveReceiptStatus::Accepted) + ); + assert!(request_path.is_file()); + + fs::remove_dir(&receipt_path).unwrap(); + assert!( + write_batch_status( + &request_dir, + &receipt_dir, + &authority, + &mut batch, + ObserveReceiptStatus::SettledUnchanged, + None, + None, + ) + .is_empty() + ); + assert_eq!( + batch.requests[0].last_status, + Some(ObserveReceiptStatus::SettledUnchanged) + ); + assert!(!request_path.exists()); + assert_eq!( + read_receipt(&receipt_dir, &request.request_id) + .unwrap() + .unwrap() + .status, + ObserveReceiptStatus::SettledUnchanged + ); + } + + #[test] + fn full_writer_queue_is_reported_without_consuming_the_frame() { + let owner = RuntimeOwner::new( + RuntimeIncarnation::new("incarnation").unwrap(), + OwnerClaim::new("claim").unwrap(), + ); + let mut lifecycle = RuntimeLifecycle::new(); + lifecycle.claim(owner.clone()); + let child = Command::new(std::env::current_exe().unwrap()) + .arg("--help") + .stdin(Stdio::null()) + .stdout(Stdio::null()) + .stderr(Stdio::null()) + .spawn() + .unwrap(); + let (writer, queued) = mpsc::sync_channel(1); + writer.try_send(vec![1]).unwrap(); + let receipt_dir = tempfile::tempdir().unwrap(); + let mut runtime = RuntimeProcess { + scheme: "dev.x".into(), + owner: owner.clone(), + lifecycle, + child, + writer: Some(writer), + writer_thread: None, + reader_thread: None, + bindings: BTreeMap::new(), + request_dir: receipt_dir.path().join("requests"), + receipt_dir: receipt_dir.path().to_path_buf(), + process_health: ResourceProfileHealth { + scheme: "dev.x".into(), + binding: None, + state: RuntimeHealthState::Starting, + detail: None, + }, + }; + let error = runtime + .send(HostMessage::Observe { + owner, + binding_id: BindingId::new("binding").unwrap(), + registration: RegistrationToken::new("registration").unwrap(), + demand_watermark: 1, + }) + .unwrap_err(); + assert!(error.to_string().contains("queue is full")); + assert_eq!(queued.try_recv().unwrap(), vec![1]); + runtime.stop(); + } + #[test] fn stale_owner_envelopes_cannot_target_a_replacement_with_the_same_runtime_key() { let old = RuntimeOwner::new( @@ -1270,7 +2207,14 @@ mod tests { generation: 1, }; let (tx, _rx) = mpsc::sync_channel(1); - let mut worker = Worker::new(PathBuf::from("/"), "host".into(), tx); + let temp = tempfile::tempdir().unwrap(); + let mut worker = Worker::new( + temp.path().to_path_buf(), + "host".into(), + tx, + temp.path().join("requests"), + temp.path().join("receipts"), + ); worker.runtimes.insert( key.clone(), RuntimeProcess { @@ -1282,6 +2226,8 @@ mod tests { writer_thread: None, reader_thread: None, bindings: BTreeMap::new(), + request_dir: temp.path().join("requests"), + receipt_dir: temp.path().join("receipts"), process_health: ResourceProfileHealth { scheme: "dev.x".into(), binding: None, diff --git a/src/telemetry.rs b/src/telemetry.rs index 3b05065b..b593555f 100644 --- a/src/telemetry.rs +++ b/src/telemetry.rs @@ -44,16 +44,25 @@ const DURATION_BUCKET_BOUNDARIES: [f64; 12] = [ /// View mapping each duration instrument onto seconds-scale explicit buckets (see /// [`DURATION_BUCKET_BOUNDARIES`]); every other instrument keeps its default aggregation. fn duration_view(instrument: &Instrument) -> Option { - match instrument.name() { - "reconcile_pass_duration_seconds" | "session_start_duration_seconds" => Stream::builder() + duration_instrument(instrument.name()).then(|| { + Stream::builder() .with_aggregation(Aggregation::ExplicitBucketHistogram { boundaries: DURATION_BUCKET_BOUNDARIES.into(), record_min_max: true, }) .build() - .ok(), - _ => None, - } + .expect("duration histogram view is valid") + }) +} + +fn duration_instrument(name: &str) -> bool { + matches!( + name, + "reconcile_pass_duration_seconds" + | "session_start_duration_seconds" + | "resource_observe_dispatch_seconds" + | "resource_observe_settle_seconds" + ) } /// Level filtering for the stderr fmt layer, defaulting to INFO. `RUST_LOG` overrides it on that @@ -297,3 +306,21 @@ fn build_log_exporter() -> Result]` (reads new content from stdin) - `st2 context append [] --decision "" --why ""` -Declared Resource bindings (`resource` nodes in your own declaration; writes republish it under CAS): +Declared Resource bindings (`resource` nodes in your own declaration; authoring writes republish under CAS): - `st2 resource ls [] [--json]` · `st2 resource read [] [--json]` +- `st2 resource refresh [--agent ] [--wait ] [--json]` · `st2 resource refresh [--wait ] [--json]` (one positional targets the caller; use either a leading identity or `--agent`, never both; demand observation never rewrites the declaration) - `st2 resource add --uri --reason [--inactive-reason ]` - `st2 resource remove ` · `st2 resource rename ` - *writes also take `--agent ` (any declaration you may publish) and `--json`* diff --git a/tests/agent_resource.rs b/tests/agent_resource.rs index 91a9cb53..fb043822 100644 --- a/tests/agent_resource.rs +++ b/tests/agent_resource.rs @@ -1,10 +1,14 @@ //! `st2 resource` operates on declared Agent Spec Resource bindings: `ls`/`read` project them, -//! and `add`/`remove`/`rename` mutate one binding through mediated CAS publication without the -//! caller rendering KDL. +//! `refresh` requests a demand observation without rewriting the declaration, and +//! `add`/`remove`/`rename` mutate one binding through mediated CAS publication. use std::fs; use std::path::Path; -use std::process::Command; +use std::process::{Command, Stdio}; +use std::sync::Mutex; +use std::time::{Duration, Instant}; + +static OBSERVE_ENV_LOCK: Mutex<()> = Mutex::new(()); fn write(root: &Path, relative: &str, contents: &str) { let path = root.join(relative); @@ -115,7 +119,10 @@ fn ls_json_and_read_expose_every_declared_field() { // already emits, and INVARIANTS.md pins that surface to preserve its field names. The // snake_case is inconsistent with sibling roster fields but predates this change. assert_eq!(row["inactive_reason"], "Superseded by #43."); - assert_eq!(row["selector"], serde_json::json!({"topics": ["ci.failure"]})); + assert_eq!( + row["selector"], + serde_json::json!({"topics": ["ci.failure"]}) + ); let read = ok(root, &["resource", "read", "worker", "work"]); assert!( @@ -167,7 +174,8 @@ fn add_publishes_one_binding_and_is_idempotent_on_identical_bytes() { let after = spec(root); assert!(after.contains("resource \"work\""), "got:\n{after}"); assert!( - after.contains(r####"selector=###"{"literal":"a\"#b\"##c","topics":["ci.failure"]}"###"####), + after + .contains(r####"selector=###"{"literal":"a\"#b\"##c","topics":["ci.failure"]}"###"####), "selector must use the smallest safe raw-string fence:\n{after}" ); assert!( @@ -366,8 +374,15 @@ fn mutation_refuses_an_invalid_uri_an_empty_reason_and_a_nix_managed_declaration let reserved = run( root, &[ - "resource", "add", "declaration", "--agent", "worker", "--uri", - "https://example.test/x", "--reason", "Probe.", + "resource", + "add", + "declaration", + "--agent", + "worker", + "--uri", + "https://example.test/x", + "--reason", + "Probe.", ], ); assert!( @@ -454,10 +469,15 @@ fn a_binding_with_a_trailing_line_comment_is_removable_and_updatable() { ok( root, &[ - "resource", "add", "work", - "--agent", "worker", - "--uri", "https://example.test/x", - "--reason", "Changed.", + "resource", + "add", + "work", + "--agent", + "worker", + "--uri", + "https://example.test/x", + "--reason", + "Changed.", ], ); let updated = spec(root); @@ -486,3 +506,372 @@ fn a_binding_with_a_trailing_line_comment_is_removable_and_updatable() { "unrelated bytes must survive:\n{after}" ); } + +#[test] +fn refresh_binding_validation_and_generation_share_one_catalog_snapshot() { + let _guard = OBSERVE_ENV_LOCK + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + let temporary = tempfile::tempdir().unwrap(); + let root = temporary.path().join("catalog"); + let state = temporary.path().join("state"); + write( + &root, + "h/worker/agent.kdl", + &declaration( + "worker", + "catalog", + " resource \"work\" reason=\"Old binding.\" uri=\"https://example.test/old\"\n", + ), + ); + ok( + &root, + &[ + "resource", + "add", + "anchor", + "--agent", + "worker", + "--uri", + "https://example.test/anchor", + "--reason", + "Initialize the catalog generation.", + ], + ); + let old_generation = st2::resource_observe::catalog_generation(&root) + .unwrap() + .expect("mediated edit initialized catalog generation"); + + let previous_state = std::env::var_os("XDG_STATE_HOME"); + unsafe { std::env::set_var("XDG_STATE_HOME", &state) }; + st2::event::publish_owner_binding_for_test(&root, "h").unwrap(); + let scope = st2::park::SupervisorScope::current(&root, "h").unwrap(); + let request_dir = scope + .park_dir() + .parent() + .unwrap() + .join("observe-requests"); + match previous_state { + Some(value) => unsafe { std::env::set_var("XDG_STATE_HOME", value) }, + None => unsafe { std::env::remove_var("XDG_STATE_HOME") }, + } + + let snapshot_ready = temporary.path().join("snapshot-ready"); + let snapshot_release = temporary.path().join("snapshot-release"); + let replacement_attempt = temporary.path().join("replacement-attempt"); + let refresh = Command::new(env!("CARGO_BIN_EXE_st2")) + .args(["--catalog", root.to_str().unwrap()]) + .args([ + "resource", "refresh", "worker", "work", "--wait", "0", "--host", "h", + ]) + .env("XDG_STATE_HOME", &state) + .env("ST2_TEST_RESOURCE_REFRESH_SNAPSHOT_READY", &snapshot_ready) + .env( + "ST2_TEST_RESOURCE_REFRESH_SNAPSHOT_RELEASE", + &snapshot_release, + ) + .env_remove("ST_AGENT") + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .spawn() + .unwrap(); + wait_for_file(&snapshot_ready); + + let mut replacement = Command::new(env!("CARGO_BIN_EXE_st2")) + .args(["--catalog", root.to_str().unwrap()]) + .args([ + "resource", + "add", + "work", + "--agent", + "worker", + "--uri", + "https://example.test/replacement", + "--reason", + "Replacement binding.", + ]) + .env("ST2_TEST_CATALOG_LOCK_ATTEMPT", &replacement_attempt) + .env_remove("ST_AGENT") + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .spawn() + .unwrap(); + wait_for_file(&replacement_attempt); + assert!( + replacement.try_wait().unwrap().is_none(), + "replacement crossed the shared snapshot lock before generation capture" + ); + assert!( + spec(&root).contains("https://example.test/old"), + "replacement landed while refresh held its snapshot" + ); + + fs::write(&snapshot_release, b"release").unwrap(); + let refresh = refresh.wait_with_output().unwrap(); + assert!( + !refresh.status.success(), + "zero client wait should leave the request queued" + ); + let replacement = replacement.wait_with_output().unwrap(); + assert!( + replacement.status.success(), + "stdout: {}\nstderr: {}", + stdout(&replacement), + stderr(&replacement) + ); + + let request = wait_for_refresh_request(&request_dir); + assert_eq!( + request["expectedCatalogGeneration"], + serde_json::json!(old_generation), + "request generation must come from the validated old binding snapshot" + ); + let replacement_generation = st2::resource_observe::catalog_generation(&root) + .unwrap() + .unwrap(); + assert!( + replacement_generation > old_generation, + "replacement did not advance catalog generation" + ); + assert!(spec(&root).contains("https://example.test/replacement")); +} + +#[test] +fn refresh_cli_reports_exact_receipts_and_wait_expiry_keeps_the_request() { + let _guard = OBSERVE_ENV_LOCK + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + let temporary = tempfile::tempdir().unwrap(); + let root = temporary.path().join("catalog"); + let state = temporary.path().join("state"); + write( + &root, + "h/worker/agent.kdl", + &declaration( + "worker", + "catalog", + " resource \"work\" reason=\"Observed.\" uri=\"dev.example://work\"\n", + ), + ); + + let previous_state = std::env::var_os("XDG_STATE_HOME"); + unsafe { std::env::set_var("XDG_STATE_HOME", &state) }; + st2::event::publish_owner_binding_for_test(&root, "h").unwrap(); + let scope = st2::park::SupervisorScope::current(&root, "h").unwrap(); + let scope_root = scope.park_dir().parent().unwrap().to_path_buf(); + match previous_state { + Some(value) => unsafe { std::env::set_var("XDG_STATE_HOME", value) }, + None => unsafe { std::env::remove_var("XDG_STATE_HOME") }, + } + let request_dir = scope_root.join("observe-requests"); + let receipt_dir = scope_root.join("observe-receipts"); + + let unchanged = spawn_refresh(&root, &state, 2); + let request = wait_for_refresh_request(&request_dir); + publish_refresh_receipt(&receipt_dir, &request, "settledUnchanged", None); + let unchanged = unchanged.wait_with_output().unwrap(); + assert!( + unchanged.status.success(), + "stdout: {}\nstderr: {}", + stdout(&unchanged), + stderr(&unchanged) + ); + let unchanged_json: serde_json::Value = serde_json::from_slice(&unchanged.stdout).unwrap(); + assert_eq!(unchanged_json["status"], "settledUnchanged"); + assert_eq!(unchanged_json["recipient"], "h.worker"); + clear_refresh_records(&request_dir, &receipt_dir); + + let failed = spawn_refresh(&root, &state, 2); + let request = wait_for_refresh_request(&request_dir); + publish_refresh_receipt( + &receipt_dir, + &request, + "settledFailed", + Some("provider refused"), + ); + let failed = failed.wait_with_output().unwrap(); + assert!(!failed.status.success()); + let failed_json: serde_json::Value = serde_json::from_slice(&failed.stdout).unwrap(); + assert_eq!(failed_json["status"], "settledFailed"); + assert_eq!(failed_json["diagnostic"], "provider refused"); + clear_refresh_records(&request_dir, &receipt_dir); + + let timed_out = Command::new(env!("CARGO_BIN_EXE_st2")) + .args(["--catalog", root.to_str().unwrap()]) + .args([ + "resource", "refresh", "worker", "work", "--wait", "0", "--json", "--host", "h", + ]) + .env("XDG_STATE_HOME", &state) + .env_remove("ST_AGENT") + .output() + .unwrap(); + assert!(!timed_out.status.success()); + let timeout_json: serde_json::Value = serde_json::from_slice(&timed_out.stdout).unwrap(); + assert_eq!(timeout_json["status"], "timeout"); + assert_eq!(timeout_json["queued"], true); + assert!( + fs::read_dir(&request_dir) + .unwrap() + .flatten() + .any(|entry| entry + .path() + .extension() + .is_some_and(|extension| extension == "json")), + "the client wait bound dropped its queued request" + ); + clear_refresh_records(&request_dir, &receipt_dir); + + let final_read_ready = temporary.path().join("observe-final-read-ready"); + let final_read_release = temporary.path().join("observe-final-read-release"); + let final_read = Command::new(env!("CARGO_BIN_EXE_st2")) + .args(["--catalog", root.to_str().unwrap()]) + .args([ + "resource", "refresh", "worker", "work", "--wait", "1", "--json", "--host", "h", + ]) + .env("XDG_STATE_HOME", &state) + .env("ST2_TEST_OBSERVE_WAIT_TIMEOUT_READY", &final_read_ready) + .env("ST2_TEST_OBSERVE_WAIT_TIMEOUT_RELEASE", &final_read_release) + .env_remove("ST_AGENT") + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .spawn() + .unwrap(); + let request = wait_for_refresh_request(&request_dir); + wait_for_file(&final_read_ready); + publish_refresh_receipt(&receipt_dir, &request, "settledUnchanged", None); + fs::write(&final_read_release, b"release").unwrap(); + let final_read = final_read.wait_with_output().unwrap(); + assert!( + final_read.status.success(), + "stdout: {}\nstderr: {}", + stdout(&final_read), + stderr(&final_read) + ); + let final_read_json: serde_json::Value = serde_json::from_slice(&final_read.stdout).unwrap(); + assert_eq!(final_read_json["status"], "settledUnchanged"); + clear_refresh_records(&request_dir, &receipt_dir); + + let no_supervisor_root = temporary.path().join("no-supervisor-catalog"); + write( + &no_supervisor_root, + "h/worker/agent.kdl", + &declaration( + "worker", + "catalog", + " resource \"work\" reason=\"Observed.\" uri=\"dev.example://work\"\n", + ), + ); + let no_supervisor_state = temporary.path().join("no-supervisor-state"); + let no_supervisor = Command::new(env!("CARGO_BIN_EXE_st2")) + .args(["--catalog", no_supervisor_root.to_str().unwrap()]) + .args([ + "resource", "refresh", "worker", "work", "--wait", "0", "--host", "h", + ]) + .env("XDG_STATE_HOME", no_supervisor_state) + .env_remove("ST_AGENT") + .output() + .unwrap(); + assert!(!no_supervisor.status.success()); + assert!(stderr(&no_supervisor).contains("no live Resource Profile supervisor")); +} + +fn spawn_refresh(root: &Path, state: &Path, wait: u64) -> std::process::Child { + let wait = wait.to_string(); + Command::new(env!("CARGO_BIN_EXE_st2")) + .args(["--catalog", root.to_str().unwrap()]) + .args([ + "resource", + "refresh", + "worker", + "work", + "--wait", + wait.as_str(), + "--json", + "--host", + "h", + ]) + .env("XDG_STATE_HOME", state) + .env_remove("ST_AGENT") + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .spawn() + .unwrap() +} + +fn wait_for_refresh_request(dir: &Path) -> serde_json::Value { + let deadline = Instant::now() + Duration::from_secs(5); + loop { + if let Some(request) = fs::read_dir(dir) + .into_iter() + .flatten() + .flatten() + .find_map(|entry| { + (entry + .path() + .extension() + .is_some_and(|extension| extension == "json")) + .then(|| fs::read(entry.path()).ok()) + .flatten() + .and_then(|bytes| serde_json::from_slice(&bytes).ok()) + }) + { + return request; + } + assert!( + Instant::now() < deadline, + "timed out waiting for refresh request" + ); + std::thread::yield_now(); + } +} + +fn wait_for_file(path: &Path) { + let deadline = Instant::now() + Duration::from_secs(5); + while !path.is_file() { + assert!( + Instant::now() < deadline, + "timed out waiting for {}", + path.display() + ); + std::thread::yield_now(); + } +} + +fn publish_refresh_receipt( + dir: &Path, + request: &serde_json::Value, + status: &str, + diagnostic: Option<&str>, +) { + fs::create_dir_all(dir).unwrap(); + let request_id = request["requestId"].as_str().unwrap(); + let mut receipt = serde_json::json!({ + "schema": "st2.resource-observe-receipt.v1", + "requestId": request_id, + "recipient": request["recipient"], + "binding": request["binding"], + "status": status, + "authority": { + "owner": {"incarnation": "incarnation", "claim": "claim"}, + "bindingId": "binding", + "registration": "registration" + }, + "demandWatermark": 1, + "updatedAt": "2026-08-31T00:00:00Z" + }); + if let Some(diagnostic) = diagnostic { + receipt["diagnostic"] = serde_json::Value::String(diagnostic.to_owned()); + } + let temporary = dir.join(format!(".{request_id}.tmp")); + let final_path = dir.join(format!("{request_id}.json")); + fs::write(&temporary, serde_json::to_vec(&receipt).unwrap()).unwrap(); + fs::rename(temporary, final_path).unwrap(); +} + +fn clear_refresh_records(request_dir: &Path, receipt_dir: &Path) { + for dir in [request_dir, receipt_dir] { + for entry in fs::read_dir(dir).into_iter().flatten().flatten() { + fs::remove_file(entry.path()).unwrap(); + } + } +} diff --git a/tests/resource_profile_supervisor_e2e.rs b/tests/resource_profile_supervisor_e2e.rs index 8a980597..9cbca3cd 100755 --- a/tests/resource_profile_supervisor_e2e.rs +++ b/tests/resource_profile_supervisor_e2e.rs @@ -6,16 +6,23 @@ use std::io::Write as _; use std::os::unix::ffi::OsStrExt as _; use std::os::unix::fs::PermissionsExt as _; use std::path::{Path, PathBuf}; +use std::sync::Mutex; use std::time::{Duration, Instant}; +use st2::resource_observe::{ + MAX_PENDING_OBSERVE_REQUESTS, ObserveAdmissionBackpressure, ObserveReceiptStatus, + ObserveRequest, +}; use st2::resource_profile::{ - BindingId, HostMessage, RegistrationToken, ResourceFact, RuntimeHealthState, RuntimeMessage, - RuntimeOwner, SnapshotBytes, decode_host_line, encode_runtime_line, + BindingId, HostMessage, ObservationResult, Publication, RegistrationToken, ResourceFact, + RuntimeHealthState, RuntimeMessage, RuntimeOwner, SnapshotBytes, SnapshotDigest, + decode_host_line, encode_runtime_line, }; use st2::resource_profile_supervisor::ResourceProfileSupervisor; const SCHEME: &str = "dev.example.observable"; const SCHEMA_ID: &str = "dev.example.observable.snapshot.v1"; +static TEST_ENV_LOCK: Mutex<()> = Mutex::new(()); const MEDIA_TYPE: &str = "application/json"; #[derive(Clone)] @@ -25,8 +32,15 @@ struct Registration { registration: RegistrationToken, } +#[derive(Clone)] +struct Observation { + registration: Registration, + demand_watermark: u64, +} + struct RuntimeControl { register_path: PathBuf, + observe_path: PathBuf, output: File, } @@ -52,6 +66,83 @@ impl RuntimeControl { }) } + fn observations(&self) -> Vec { + fs::read_to_string(&self.observe_path) + .unwrap_or_default() + .lines() + .filter_map(|line| { + let message = decode_host_line(format!("{line}\n").as_bytes()).ok()?; + let HostMessage::Observe { + owner, + binding_id, + registration, + demand_watermark, + } = message + else { + return None; + }; + Some(Observation { + registration: Registration { + owner, + binding_id, + registration, + }, + demand_watermark, + }) + }) + .collect() + } + + fn wait_for_observation(&self, count: usize) -> Observation { + wait_until("runtime observe message", || { + self.observations().get(count.saturating_sub(1)).cloned() + }) + } + + fn unchanged(&self, observation: &Observation) { + self.result(observation, ObservationResult::Unchanged); + } + + fn failed(&self, observation: &Observation, diagnostic: &str) { + self.result( + observation, + ObservationResult::Failed { + diagnostic: Some(diagnostic.to_owned()), + }, + ); + } + + fn publish_observation( + &self, + observation: &Observation, + bytes: &[u8], + topics: &[&str], + fact: &str, + ) { + self.result( + observation, + ObservationResult::Published { + publication: publication(bytes, topics, fact), + }, + ); + } + + fn result(&self, observation: &Observation, result: ObservationResult) { + let registration = &observation.registration; + let message = RuntimeMessage::ObservationResult { + owner: registration.owner.clone(), + binding_id: registration.binding_id.clone(), + registration: registration.registration.clone(), + demand_watermark: observation.demand_watermark, + result, + }; + let mut output = &self.output; + output + .write_all(&encode_runtime_line(&message).unwrap()) + .unwrap(); + output.flush().unwrap(); + } + fn publish( &self, registration: &Registration, @@ -63,14 +154,7 @@ impl RuntimeControl { owner: registration.owner.clone(), binding_id: registration.binding_id.clone(), registration: registration.registration.clone(), - schema_id: SCHEMA_ID.to_owned(), - media_type: MEDIA_TYPE.to_owned(), - bytes: SnapshotBytes::new(bytes.to_vec()).unwrap(), - topics: topics.iter().map(|topic| (*topic).to_owned()).collect(), - facts: Some(vec![ - ResourceFact::current("revision", health_marker).unwrap(), - ]), - observed_at: None, + publication: publication(bytes, topics, health_marker), }; let health = RuntimeMessage::Health { owner: registration.owner.clone(), @@ -90,6 +174,16 @@ impl RuntimeControl { } } +fn publication(bytes: &[u8], topics: &[&str], fact: &str) -> Publication { + Publication { + schema_id: SCHEMA_ID.to_owned(), + media_type: MEDIA_TYPE.to_owned(), + bytes: SnapshotBytes::new(bytes.to_vec()).unwrap(), + topics: topics.iter().map(|topic| (*topic).to_owned()).collect(), + facts: Some(vec![ResourceFact::current("revision", fact).unwrap()]), + } +} + struct CatalogFixture { root: PathBuf, host: String, @@ -121,6 +215,7 @@ impl CatalogFixture { let control_dir = root.join("runtime-control"); fs::create_dir_all(&control_dir).unwrap(); let register_path = control_dir.join("register.ndjson"); + let observe_path = control_dir.join("observe.ndjson"); let fifo_path = control_dir.join("runtime-output.fifo"); let fifo = CString::new(fifo_path.as_os_str().as_bytes()).unwrap(); assert_eq!(unsafe { libc::mkfifo(fifo.as_ptr(), 0o600) }, 0); @@ -133,7 +228,7 @@ impl CatalogFixture { let runtime = root.join("fake-observable-runtime"); fs::write( &runtime, - "#!/bin/sh\nset -eu\nIFS= read -r registration\nprintf '%s\\n' \"$registration\" > \"$1/register.ndjson\"\nexec cat \"$1/runtime-output.fifo\"\n", + "#!/bin/sh\nset -eu\nexec 3<&0\n(\n while IFS= read -r frame; do\n case \"$frame\" in\n *'\"type\":\"register\"'*) printf '%s\\n' \"$frame\" > \"$1/register.ndjson\" ;;\n *'\"type\":\"observe\"'*) printf '%s\\n' \"$frame\" >> \"$1/observe.ndjson\" ;;\n esac\n done\n) <&3 &\nexec cat \"$1/runtime-output.fifo\"\n", ) .unwrap(); fs::set_permissions(&runtime, fs::Permissions::from_mode(0o755)).unwrap(); @@ -145,6 +240,7 @@ impl CatalogFixture { class "immediate" runtime {{ argv "{}" "{}" + capability "demand" }} }} "#, @@ -160,6 +256,7 @@ impl CatalogFixture { agent_dir, runtime: RuntimeControl { register_path, + observe_path, output, }, } @@ -173,6 +270,10 @@ impl CatalogFixture { } fn refresh(&self, supervisor: &ResourceProfileSupervisor) { + self.refresh_generation(supervisor, 1); + } + + fn refresh_generation(&self, supervisor: &ResourceProfileSupervisor, generation: u64) { let (config, profiles) = st2::catalog::declared_profile_catalog(&self.root).unwrap(); let discovery = st2::discover_strict(&self.root); assert!( @@ -180,7 +281,7 @@ impl CatalogFixture { "fixture catalog must be valid: {:?}", discovery.errors ); - let report = supervisor.refresh(&config, &profiles, Some(1), &discovery.specs); + let report = supervisor.refresh(&config, &profiles, Some(generation), &discovery.specs); assert!( report.warnings.is_empty(), "Resource Profile refresh warnings: {:?}", @@ -198,10 +299,41 @@ impl CatalogFixture { .park_dir(); park_dir.parent().unwrap().join("stream-owner.json") } + + fn observe_request_dir(&self) -> PathBuf { + st2::park::SupervisorScope::current(&self.root, &self.host) + .unwrap() + .park_dir() + .parent() + .unwrap() + .join("observe-requests") + } + + fn observe_receipt_dir(&self) -> PathBuf { + st2::park::SupervisorScope::current(&self.root, &self.host) + .unwrap() + .park_dir() + .parent() + .unwrap() + .join("observe-receipts") + } + + fn request(&self) -> ObserveRequest { + ObserveRequest::new( + format!("{}.worker", self.host), + "observed".to_owned(), + Some(1), + None, + ) + .unwrap() + } } #[test] fn observable_publication_reaches_builtin_resync_with_filter_catch_up_and_scope_isolation() { + let _guard = TEST_ENV_LOCK + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); let temporary = tempfile::tempdir().unwrap(); let state = temporary.path().join("state"); unsafe { std::env::set_var("XDG_STATE_HOME", &state) }; @@ -282,7 +414,11 @@ fn observable_publication_reaches_builtin_resync_with_filter_catch_up_and_scope_ st2::event::publish_owner_binding_for_test(&primary.root, &primary.host).unwrap(); primary.refresh(&primary_supervisor); let caught_up_inbox = resync_inbox(&primary.agent_dir); - assert_eq!(caught_up_inbox.len(), 1, "supersession keeps one unread head"); + assert_eq!( + caught_up_inbox.len(), + 1, + "supersession keeps one unread head" + ); assert_ne!( caught_up_inbox, first_inbox, "restoring delivery must replace the old head with the pending digest" @@ -346,6 +482,412 @@ fn observable_publication_reaches_builtin_resync_with_filter_catch_up_and_scope_ "the surviving scope must still publish and invalidate" ); } +#[test] +fn demand_observation_settlement_matrix_is_atomic_and_preserves_facts() { + let _guard = TEST_ENV_LOCK + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + let temporary = tempfile::tempdir().unwrap(); + unsafe { std::env::set_var("XDG_STATE_HOME", temporary.path().join("state")) }; + + let fixture = CatalogFixture::new(temporary.path().join("catalog"), "alpha"); + st2::event::publish_owner_binding_for_test(&fixture.root, &fixture.host).unwrap(); + let supervisor = fixture.supervisor(); + fixture.runtime.registration(); + + let unchanged = fixture.request(); + let unchanged_client = + st2::resource_observe::submit_request(&fixture.root, &fixture.host, &unchanged).unwrap(); + let unchanged_observation = fixture.runtime.wait_for_observation(1); + assert_eq!(unchanged_observation.demand_watermark, 1); + fixture.runtime.unchanged(&unchanged_observation); + let unchanged_receipt = unchanged_client + .wait_for_terminal(Duration::from_secs(2)) + .unwrap() + .receipt + .unwrap(); + assert_eq!( + unchanged_receipt.status, + ObserveReceiptStatus::SettledUnchanged + ); + assert_eq!(unchanged_receipt.demand_watermark, Some(1)); + assert_eq!(unchanged_receipt.digest, None); + + let changed = fixture.request(); + let changed_client = + st2::resource_observe::submit_request(&fixture.root, &fixture.host, &changed).unwrap(); + let changed_observation = fixture.runtime.wait_for_observation(2); + let changed_bytes = br#"{"demand":"changed"}"#; + assert_eq!(changed_observation.demand_watermark, 2); + fixture.runtime.publish_observation( + &changed_observation, + changed_bytes, + &["selected"], + "demand-changed", + ); + let changed_receipt = changed_client + .wait_for_terminal(Duration::from_secs(2)) + .unwrap() + .receipt + .unwrap(); + assert_eq!(changed_receipt.status, ObserveReceiptStatus::SettledChanged); + assert_eq!( + changed_receipt.digest, + Some(SnapshotDigest::of(changed_bytes)) + ); + assert_eq!(fs::read(fixture.snapshot_path()).unwrap(), changed_bytes); + let changed_inbox = resync_inbox(&fixture.agent_dir); + assert_eq!(changed_inbox.len(), 1); + assert!( + changed_inbox[0].contains(r#""facts":[{"key":"revision","after":"demand-changed"}]"#), + "{}", + changed_inbox[0] + ); + + let failed = fixture.request(); + let failed_client = + st2::resource_observe::submit_request(&fixture.root, &fixture.host, &failed).unwrap(); + let failed_observation = fixture.runtime.wait_for_observation(3); + assert_eq!(failed_observation.demand_watermark, 3); + fixture + .runtime + .failed(&failed_observation, "provider refused"); + let failed_receipt = failed_client + .wait_for_terminal(Duration::from_secs(2)) + .unwrap() + .receipt + .unwrap(); + assert_eq!(failed_receipt.status, ObserveReceiptStatus::SettledFailed); + assert_eq!( + failed_receipt.diagnostic.as_deref(), + Some("provider refused") + ); + drop(supervisor); +} + +#[test] +fn demand_observation_coalesces_and_fences_watermarks() { + let _guard = TEST_ENV_LOCK + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + let temporary = tempfile::tempdir().unwrap(); + unsafe { std::env::set_var("XDG_STATE_HOME", temporary.path().join("state")) }; + + let fixture = CatalogFixture::new(temporary.path().join("catalog"), "alpha"); + st2::event::publish_owner_binding_for_test(&fixture.root, &fixture.host).unwrap(); + let supervisor = fixture.supervisor(); + fixture.runtime.registration(); + + let leading = fixture.request(); + let leading_client = + st2::resource_observe::submit_request(&fixture.root, &fixture.host, &leading).unwrap(); + let leading_observation = fixture.runtime.wait_for_observation(1); + let trailing_a = fixture.request(); + let trailing_a_path = fixture + .observe_request_dir() + .join(format!("{}.json", trailing_a.request_id)); + let trailing_a_client = + st2::resource_observe::submit_request(&fixture.root, &fixture.host, &trailing_a).unwrap(); + let trailing_b = fixture.request(); + let trailing_b_path = fixture + .observe_request_dir() + .join(format!("{}.json", trailing_b.request_id)); + let trailing_b_client = + st2::resource_observe::submit_request(&fixture.root, &fixture.host, &trailing_b).unwrap(); + wait_until("trailing request retention", || { + (trailing_a_path.exists() && trailing_b_path.exists()).then_some(()) + }); + assert_eq!( + fixture.runtime.observations().len(), + 1, + "one in-flight demand permits only one coalesced trailing batch" + ); + assert_eq!(leading_observation.demand_watermark, 1); + fixture.runtime.unchanged(&leading_observation); + let trailing_observation = fixture.runtime.wait_for_observation(2); + assert_eq!(trailing_observation.demand_watermark, 2); + fixture.runtime.unchanged(&trailing_observation); + assert_eq!( + leading_client + .wait_for_terminal(Duration::from_secs(2)) + .unwrap() + .receipt + .unwrap() + .demand_watermark, + Some(1) + ); + for client in [trailing_a_client, trailing_b_client] { + assert_eq!( + client + .wait_for_terminal(Duration::from_secs(2)) + .unwrap() + .receipt + .unwrap() + .demand_watermark, + Some(2) + ); + } + wait_until("durable request cleanup after terminal receipts", || { + (!trailing_a_path.exists() && !trailing_b_path.exists()).then_some(()) + }); + + let mismatched = fixture.request(); + let mismatched_client = + st2::resource_observe::submit_request(&fixture.root, &fixture.host, &mismatched).unwrap(); + let mut mismatched_observation = fixture.runtime.wait_for_observation(3); + assert_eq!(mismatched_observation.demand_watermark, 3); + mismatched_observation.demand_watermark = 2; + fixture.runtime.unchanged(&mismatched_observation); + let mismatched_receipt = mismatched_client + .wait_for_terminal(Duration::from_secs(2)) + .unwrap() + .receipt + .unwrap(); + assert_eq!( + mismatched_receipt.status, + ObserveReceiptStatus::ProviderUnavailable, + "a stale watermark must fail the runtime rather than settle the current demand" + ); + + fixture.refresh(&supervisor); + + let stale = fixture.request(); + let stale_client = + st2::resource_observe::submit_request(&fixture.root, &fixture.host, &stale).unwrap(); + fixture.runtime.wait_for_observation(4); + fixture.refresh_generation(&supervisor, 2); + let stale_receipt = stale_client + .wait_for_terminal(Duration::from_secs(2)) + .unwrap() + .receipt + .unwrap(); + assert_eq!(stale_receipt.status, ObserveReceiptStatus::StaleGeneration); +} + +#[test] +fn demand_observation_survives_restart_disconnect_and_denies_missing_capability() { + let _guard = TEST_ENV_LOCK + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + let temporary = tempfile::tempdir().unwrap(); + unsafe { std::env::set_var("XDG_STATE_HOME", temporary.path().join("state")) }; + + let fixture = CatalogFixture::new(temporary.path().join("catalog"), "alpha"); + st2::event::publish_owner_binding_for_test(&fixture.root, &fixture.host).unwrap(); + let initial = fixture.supervisor(); + fixture.runtime.registration(); + let restart_request = fixture.request(); + let restart_path = fixture + .observe_request_dir() + .join(format!("{}.json", restart_request.request_id)); + let restart_client = + st2::resource_observe::submit_request(&fixture.root, &fixture.host, &restart_request) + .unwrap(); + let interrupted_observation = fixture.runtime.wait_for_observation(1); + assert!( + restart_path.is_file(), + "enqueue must not remove the restart-recoverable durable request" + ); + drop(initial); + let restarted = fixture.supervisor(); + let restart_observation = fixture.runtime.wait_for_observation(2); + assert_eq!(restart_observation.demand_watermark, 1); + assert_ne!( + restart_observation.registration.owner, interrupted_observation.registration.owner, + "restart recovery must redispatch through the new runtime owner" + ); + fixture.runtime.unchanged(&restart_observation); + assert_eq!( + restart_client + .wait_for_terminal(Duration::from_secs(2)) + .unwrap() + .receipt + .unwrap() + .status, + ObserveReceiptStatus::SettledUnchanged + ); + + let disconnected = fixture.request(); + let disconnected_id = disconnected.request_id.clone(); + let disconnected_client = + st2::resource_observe::submit_request(&fixture.root, &fixture.host, &disconnected).unwrap(); + let disconnected_observation = fixture.runtime.wait_for_observation(3); + drop(disconnected_client); + assert_eq!(disconnected_observation.demand_watermark, 2); + fixture.runtime.unchanged(&disconnected_observation); + let disconnected_receipt = wait_until("receipt after client disconnect", || { + st2::resource_observe::read_receipt(&fixture.observe_receipt_dir(), &disconnected_id) + .ok() + .flatten() + .filter(|receipt| receipt.status.is_terminal()) + }); + assert_eq!( + disconnected_receipt.status, + ObserveReceiptStatus::SettledUnchanged + ); + + let config_path = st2::catalog::config_path(&fixture.root); + let without_demand = fs::read_to_string(&config_path) + .unwrap() + .replace(" capability \"demand\"\n", ""); + fs::write(&config_path, without_demand).unwrap(); + fixture.refresh_generation(&restarted, 2); + let gated = ObserveRequest::new( + format!("{}.worker", fixture.host), + "observed".to_owned(), + None, + None, + ) + .unwrap(); + let before_gate = fixture.runtime.observations().len(); + let gated_receipt = st2::resource_observe::submit_request(&fixture.root, &fixture.host, &gated) + .unwrap() + .wait_for_terminal(Duration::from_secs(2)) + .unwrap() + .receipt + .unwrap(); + assert_eq!(gated_receipt.status, ObserveReceiptStatus::AbsentBinding); + assert_eq!( + fixture.runtime.observations().len(), + before_gate, + "a runtime without declared demand capability received an observe frame" + ); +} + +#[test] +fn durable_observe_admission_rejects_the_concurrent_257th_request() { + let _guard = TEST_ENV_LOCK + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + let temporary = tempfile::tempdir().unwrap(); + unsafe { std::env::set_var("XDG_STATE_HOME", temporary.path().join("state")) }; + + let fixture = CatalogFixture::new(temporary.path().join("catalog"), "alpha"); + st2::event::publish_owner_binding_for_test(&fixture.root, &fixture.host).unwrap(); + let supervisor = fixture.supervisor(); + fixture.runtime.registration(); + drop(supervisor); + + let submissions = (0..=MAX_PENDING_OBSERVE_REQUESTS) + .map(|_| { + let root = fixture.root.clone(); + let host = fixture.host.clone(); + std::thread::spawn(move || { + let request = ObserveRequest::new( + format!("{host}.worker"), + "observed".to_owned(), + Some(1), + None, + ) + .unwrap(); + st2::resource_observe::submit_request(&root, &host, &request) + .map(|_| true) + .map_err(|error| { + error + .downcast_ref::() + .is_some() + }) + }) + }) + .collect::>(); + let results = submissions + .into_iter() + .map(|submission| submission.join().unwrap()) + .collect::>(); + assert_eq!(results.iter().filter(|result| result.is_ok()).count(), 256); + assert_eq!( + results + .iter() + .filter(|result| matches!(result, Err(true))) + .count(), + 1, + "the only failed submission must expose structured durable backpressure" + ); + let durable_count = fs::read_dir(fixture.observe_request_dir()) + .unwrap() + .flatten() + .filter(|entry| entry.path().extension().is_some_and(|ext| ext == "json")) + .count(); + assert_eq!(durable_count, MAX_PENDING_OBSERVE_REQUESTS); +} + +#[test] +fn newer_client_generation_stays_queued_until_the_supervisor_refreshes() { + let _guard = TEST_ENV_LOCK + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + let temporary = tempfile::tempdir().unwrap(); + unsafe { std::env::set_var("XDG_STATE_HOME", temporary.path().join("state")) }; + + let fixture = CatalogFixture::new(temporary.path().join("catalog"), "alpha"); + st2::event::publish_owner_binding_for_test(&fixture.root, &fixture.host).unwrap(); + let supervisor = fixture.supervisor(); + fixture.runtime.registration(); + + let request = ObserveRequest::new( + format!("{}.worker", fixture.host), + "observed".to_owned(), + Some(2), + None, + ) + .unwrap(); + let client = + st2::resource_observe::submit_request(&fixture.root, &fixture.host, &request).unwrap(); + fixture.refresh_generation(&supervisor, 1); + let _ = supervisor.health(); + assert!( + fixture.runtime.observations().is_empty(), + "a client-ahead generation was dispatched against an older resident catalog" + ); + assert!( + st2::resource_observe::read_receipt(&fixture.observe_receipt_dir(), &request.request_id) + .unwrap() + .is_none(), + "a client-ahead generation was incorrectly terminalized as stale" + ); + + fixture.refresh_generation(&supervisor, 2); + let observation = fixture.runtime.wait_for_observation(1); + fixture.runtime.unchanged(&observation); + let receipt = client + .wait_for_terminal(Duration::from_secs(2)) + .unwrap() + .receipt + .unwrap(); + assert_eq!(receipt.status, ObserveReceiptStatus::SettledUnchanged); +} + +#[test] +fn demanded_publication_settles_changed_before_resync_delivery_failure() { + let _guard = TEST_ENV_LOCK + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + let temporary = tempfile::tempdir().unwrap(); + unsafe { std::env::set_var("XDG_STATE_HOME", temporary.path().join("state")) }; + + let fixture = CatalogFixture::new(temporary.path().join("catalog"), "alpha"); + st2::event::publish_owner_binding_for_test(&fixture.root, &fixture.host).unwrap(); + let _supervisor = fixture.supervisor(); + fixture.runtime.registration(); + + let request = fixture.request(); + let client = + st2::resource_observe::submit_request(&fixture.root, &fixture.host, &request).unwrap(); + let observation = fixture.runtime.wait_for_observation(1); + fs::remove_file(fixture.owner_binding_path()).unwrap(); + let published = br#"{"demand":"accepted-before-delivery"}"#; + fixture + .runtime + .publish_observation(&observation, published, &["selected"], "delivery-failed"); + let receipt = client + .wait_for_terminal(Duration::from_secs(2)) + .unwrap() + .receipt + .unwrap(); + assert_eq!(receipt.status, ObserveReceiptStatus::SettledChanged); + assert_eq!(receipt.digest, Some(SnapshotDigest::of(published))); + assert_eq!(fs::read(fixture.snapshot_path()).unwrap(), published); +} fn wait_for_health(supervisor: &ResourceProfileSupervisor, marker: &str) { wait_until(marker, || { @@ -397,7 +939,10 @@ fn file_tree(root: &Path) -> Vec<(PathBuf, Vec)> { if file_type.is_dir() { visit(base, &path, files); } else if file_type.is_file() { - files.push((path.strip_prefix(base).unwrap().to_path_buf(), fs::read(path).unwrap())); + files.push(( + path.strip_prefix(base).unwrap().to_path_buf(), + fs::read(path).unwrap(), + )); } } }