Skip to content

test(supervisor): give the poison window and restart backoff a clock seam#434

Open
lgahdl wants to merge 1 commit into
nullislabs:developfrom
bleu:test/284-supervisor-clock-seam
Open

test(supervisor): give the poison window and restart backoff a clock seam#434
lgahdl wants to merge 1 commit into
nullislabs:developfrom
bleu:test/284-supervisor-clock-seam

Conversation

@lgahdl

@lgahdl lgahdl commented Jul 16, 2026

Copy link
Copy Markdown
Collaborator

Summary

Fixes #284. restart_flaky_module_recovers_after_backoff, poison_pill_quarantines_module_after_threshold, and multi_chain_poisoned_module_does_not_affect_other_chains drove the supervisor's backoff/poison-window timing by sleeping real wall-clock time (~4.3s combined). The existing WasiClockOverride/ManualClock seam only virtualizes guest-visible time; the supervisor's own next_attempt/failure_timestamps bookkeeping always read the real host Instant.

Added:

  • SupervisorClock trait (supervisor.rs) — real host clock by default via a private SystemClock — threaded as a new optional constructor param through Supervisor::boot/boot_single and the full RuntimeBuilder chain (builder.rs), mirroring exactly how WasiClockOverride is already threaded (same positioning, same Option<...>-defaults-to-real shape).
  • Routed the 5 in-scope Instant::now() call sites through it: dispatch_block, dispatch_chain_log, the post-trap and post-failed-restart backoff calculations, and record_failure_and_maybe_poison. Left the dispatch latency timer (start.elapsed() in dispatch_to) on the real clock — that's instrumentation, not scheduling, and out of the issue's stated scope.
  • ManualInstantClock (test_utils/clock.rs) — a manually-advanced fake, seeded at a real Instant::now() since std::time::Instant has no arbitrary constructor — wired into the three affected tests plus TestRuntimeBuilder/TestRuntime (test_utils/harness.rs) as supervisor_clock(), mirroring the existing clock() accessor for guest time.

Notes:

  • ManualInstantClock uses std::sync::Mutex explicitly (independent of whatever Mutex import happens to be in scope elsewhere in the file) so this PR is fully decoupled from perf: migrate std::sync locks to parking_lot where not held across await #280 (the parking_lot migration) and lands cleanly regardless of merge order.
  • crates/nexum-runtime/src/runtime/{poison_policy,restart_policy}.rs, named in the issue's file list, needed no changes — both are already pure functions (should_poison(policy, recent_failures), backoff_for(failure_count)) with no clock access of their own; every actual Instant::now() call lived in supervisor.rs.
  • The restart-eligibility check is next_attempt <= now (inclusive), so each rewritten test advances the clock by exactly the backoff duration — no more sleep, no more jitter fudge factor.

Test plan

  • cargo test -p nexum-runtime --lib -- restart_flaky_module_recovers_after_backoff poison_pill_quarantines_module_after_threshold multi_chain_poisoned_module_does_not_affect_other_chains — all 3 pass in 0.71s total (previously ~4.3s of pure sleep alone, plus dispatch overhead)
  • cargo build --release --target wasm32-wasip2 for all 12 guest modules — clean
  • cargo clippy --workspace --all-targets --all-features --locked -- -D warnings — clean
  • cargo fmt --all -- --check — clean
  • cargo doc --workspace --no-deps --locked (with RUSTDOCFLAGS=-D warnings) — clean
  • cargo test --workspace --all-features --locked — full suite green

@lgahdl
lgahdl requested a review from mfw78 as a code owner July 16, 2026 17:19
…seam

poison_pill_quarantines_module_after_threshold,
restart_flaky_module_recovers_after_backoff, and
multi_chain_poisoned_module_does_not_affect_other_chains drove the
supervisor's backoff/poison-window timing by sleeping real wall-clock
time (up to ~4s combined). The existing WasiClockOverride/ManualClock
seam only virtualizes GUEST-visible time; the supervisor's own
next_attempt/failure_timestamps bookkeeping always read the real host
Instant.

Added a SupervisorClock trait (supervisor.rs) - real host clock by
default via SystemClock - threaded as a new optional constructor
param through Supervisor::boot/boot_single and the full
RuntimeBuilder chain (builder.rs), mirroring exactly how
WasiClockOverride is already threaded. Routed the 5 in-scope
Instant::now() call sites (dispatch_block, dispatch_chain_log, the
post-trap and post-failed-restart backoff calculations, and
record_failure_and_maybe_poison) through it; left the dispatch
latency timer alone, since that's instrumentation, not scheduling,
and out of the issue's stated scope.

Added ManualInstantClock (test_utils/clock.rs) - a manually-advanced
fake, seeded at a real Instant::now() since std::time::Instant has no
arbitrary constructor - and wired it into the three affected tests
plus TestRuntimeBuilder/TestRuntime (test_utils/harness.rs) as
supervisor_clock(), mirroring the existing clock() accessor. Uses
std::sync::Mutex explicitly, independent of whichever Mutex import is
in scope elsewhere in the file, so this lands cleanly regardless of

The eligibility check is next_attempt <= now (inclusive), so each
rewritten test advances the clock by exactly the backoff duration -
no more real sleeps, no more jitter fudge factor.

crates/nexum-runtime/src/runtime/{poison_policy,restart_policy}.rs,
named in the issue's file list, needed no changes: both are already
pure functions with no clock access of their own; the actual
Instant::now() calls all lived in supervisor.rs.

Fixes nullislabs#284.
@lgahdl
lgahdl force-pushed the test/284-supervisor-clock-seam branch from e783167 to ec50412 Compare July 17, 2026 14:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test: give the supervisor's poison window and restart backoff a clock seam

1 participant