test(supervisor): give the poison window and restart backoff a clock seam#434
Open
lgahdl wants to merge 1 commit into
Open
test(supervisor): give the poison window and restart backoff a clock seam#434lgahdl wants to merge 1 commit into
lgahdl wants to merge 1 commit into
Conversation
…seam
poison_pill_quarantines_module_after_threshold,
restart_flaky_module_recovers_after_backoff, and
multi_chain_poisoned_module_does_not_affect_other_chains drove the
supervisor's backoff/poison-window timing by sleeping real wall-clock
time (up to ~4s combined). The existing WasiClockOverride/ManualClock
seam only virtualizes GUEST-visible time; the supervisor's own
next_attempt/failure_timestamps bookkeeping always read the real host
Instant.
Added a SupervisorClock trait (supervisor.rs) - real host clock by
default via SystemClock - threaded as a new optional constructor
param through Supervisor::boot/boot_single and the full
RuntimeBuilder chain (builder.rs), mirroring exactly how
WasiClockOverride is already threaded. Routed the 5 in-scope
Instant::now() call sites (dispatch_block, dispatch_chain_log, the
post-trap and post-failed-restart backoff calculations, and
record_failure_and_maybe_poison) through it; left the dispatch
latency timer alone, since that's instrumentation, not scheduling,
and out of the issue's stated scope.
Added ManualInstantClock (test_utils/clock.rs) - a manually-advanced
fake, seeded at a real Instant::now() since std::time::Instant has no
arbitrary constructor - and wired it into the three affected tests
plus TestRuntimeBuilder/TestRuntime (test_utils/harness.rs) as
supervisor_clock(), mirroring the existing clock() accessor. Uses
std::sync::Mutex explicitly, independent of whichever Mutex import is
in scope elsewhere in the file, so this lands cleanly regardless of
The eligibility check is next_attempt <= now (inclusive), so each
rewritten test advances the clock by exactly the backoff duration -
no more real sleeps, no more jitter fudge factor.
crates/nexum-runtime/src/runtime/{poison_policy,restart_policy}.rs,
named in the issue's file list, needed no changes: both are already
pure functions with no clock access of their own; the actual
Instant::now() calls all lived in supervisor.rs.
Fixes nullislabs#284.
lgahdl
force-pushed
the
test/284-supervisor-clock-seam
branch
from
July 17, 2026 14:22
e783167 to
ec50412
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #284.
restart_flaky_module_recovers_after_backoff,poison_pill_quarantines_module_after_threshold, andmulti_chain_poisoned_module_does_not_affect_other_chainsdrove the supervisor's backoff/poison-window timing by sleeping real wall-clock time (~4.3s combined). The existingWasiClockOverride/ManualClockseam only virtualizes guest-visible time; the supervisor's ownnext_attempt/failure_timestampsbookkeeping always read the real hostInstant.Added:
SupervisorClocktrait (supervisor.rs) — real host clock by default via a privateSystemClock— threaded as a new optional constructor param throughSupervisor::boot/boot_singleand the fullRuntimeBuilderchain (builder.rs), mirroring exactly howWasiClockOverrideis already threaded (same positioning, sameOption<...>-defaults-to-real shape).Instant::now()call sites through it:dispatch_block,dispatch_chain_log, the post-trap and post-failed-restart backoff calculations, andrecord_failure_and_maybe_poison. Left the dispatch latency timer (start.elapsed()indispatch_to) on the real clock — that's instrumentation, not scheduling, and out of the issue's stated scope.ManualInstantClock(test_utils/clock.rs) — a manually-advanced fake, seeded at a realInstant::now()sincestd::time::Instanthas no arbitrary constructor — wired into the three affected tests plusTestRuntimeBuilder/TestRuntime(test_utils/harness.rs) assupervisor_clock(), mirroring the existingclock()accessor for guest time.Notes:
ManualInstantClockusesstd::sync::Mutexexplicitly (independent of whateverMuteximport happens to be in scope elsewhere in the file) so this PR is fully decoupled from perf: migrate std::sync locks to parking_lot where not held across await #280 (theparking_lotmigration) and lands cleanly regardless of merge order.crates/nexum-runtime/src/runtime/{poison_policy,restart_policy}.rs, named in the issue's file list, needed no changes — both are already pure functions (should_poison(policy, recent_failures),backoff_for(failure_count)) with no clock access of their own; every actualInstant::now()call lived insupervisor.rs.next_attempt <= now(inclusive), so each rewritten test advances the clock by exactly the backoff duration — no more sleep, no more jitter fudge factor.Test plan
cargo test -p nexum-runtime --lib -- restart_flaky_module_recovers_after_backoff poison_pill_quarantines_module_after_threshold multi_chain_poisoned_module_does_not_affect_other_chains— all 3 pass in 0.71s total (previously ~4.3s of pure sleep alone, plus dispatch overhead)cargo build --release --target wasm32-wasip2for all 12 guest modules — cleancargo clippy --workspace --all-targets --all-features --locked -- -D warnings— cleancargo fmt --all -- --check— cleancargo doc --workspace --no-deps --locked(withRUSTDOCFLAGS=-D warnings) — cleancargo test --workspace --all-features --locked— full suite green