test(integration): replay mainnet events and reconstruct state

[gregorydemay]

Adds the replay companion to the merged mainnet upgrade test: it takes the mainnet event log and proves the current code reconstructs exactly the state captured in the mainnet snapshot, guarding against silent decode/replay or CBOR/Candid back-compat regressions on upgrade.

• Replays the pinned mainnet snapshot event log through the existing replay path and confirms it does not panic.
• Asserts the reconstructed persisted collections equal the snapshot: token balances, order history, per-user orders, trades, trades-by-user, the user registry, and the trading-account registries.
• Loads the snapshot into PocketIC and, for every trading pair, checks the reconstructed order-book depth equals what the live snapshot canister reports.
• Reuses the snapshot download/verify flow pinned by SHA-256 and rejected on mismatch, hoisted into the shared test crate.

[Copilot]

Pull request overview

Adds an integration-test companion that replays the pinned mainnet event log with current code and verifies both persisted collections and reconstructed order books match the pinned mainnet snapshot, strengthening upgrade/back-compat regression coverage.

Changes:

• Add replay_mainnet.rs integration test that replays snapshot events and asserts reconstructed state matches snapshot + PocketIC order-book depth.
• Hoist snapshot download/extract/verify helpers + constants into the shared integration_tests crate for reuse across integration tests.
• Expose stable-collection iterators in the canister library to enable deterministic snapshot-vs-replay comparisons.

Reviewed changes

Copilot reviewed 10 out of 11 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
integration_tests/tests/replay_mainnet.rs	New integration test that replays snapshot events and cross-checks reconstructed state and order books.
integration_tests/tests/mainnet_snapshot.rs	Switches to shared snapshot helpers/constants from oisy_trade_int_tests.
integration_tests/src/lib.rs	Adds reusable snapshot download/extract/verify helpers and related constants.
integration_tests/Cargo.toml	Adds dev-deps needed for replay test (ic-stable-structures, oisy_trade_canister).
docs/src/development/specs/DEFI-2942-replay-mainnet-events.md	Spec documenting requirements and implementation approach for the replay test.
Cargo.lock	Locks new dev-dependency edges for integration tests.
canister/src/user/mod.rs	Makes registry iterators public for snapshot comparisons (needs fix for key extraction).
canister/src/order/trades/mod.rs	Adds public iterators over trades primary store and per-user index.
canister/src/order/history/mod.rs	Adds public iterators over orders primary store and per-user index.
canister/src/history/mod.rs	Adds reusable iterator helpers (iter_primary, iter_by_user) on shared History core.
canister/src/balance/token.rs	Makes balance iterator public for snapshot comparisons.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[gregorydemay]

🧐 VERDICT: CHANGES_REQUESTED — 0 blockers, 2 mediums, 1 nit; CI pending (hard block on READY).

Review details

CI: unit-tests, lint, benchmark, reproducible-build all still pending — independent hard block on a READY verdict.

Requirements

• R1 (replay does not panic): met — events collected from regions 0/1 and fed to replay_events; log asserted non-empty.
• R2 (persisted collections equal): met and genuine. All eight collections compared entry-by-entry via assert_entries_eq (length check + per-index assert_eq, no always-true assertions); both sides iterate the same key-ordered iter* accessors. Fee-pool nuance handled correctly: the fee pool is heap-only, never in the balances stable map, so TokenBalance::iter() excludes it symmetrically on both sides. R2 is not vacuous — the snapshot side is ground-truth read from stable_memory.bin, so a reconstructed-empty regression fails the length check.
• R3 (order-book depth cross-check): real comparison (reconstructed State::order_book levels vs. get_order_book_depth from the loaded snapshot canister, same MAX_DEPTH_LIMIT) BUT can pass vacuously — it loops over state.trading_pairs(), keys derived from the reconstructed state, with no non-empty guard. See inline 🟠.
• R4 (SHA-256 pin): met — download/verify hoisted verbatim into lib.rs, rejects on mismatch, removes the cached archive.

Maintainability

• Duplication: one 🟠 — the ~28-line PocketIC snapshot-load block is a near-verbatim copy across mainnet_snapshot.rs and replay_mainnet.rs; hoist a shared load_snapshot_into_pocketic helper (inline comment). No other in-diff duplication; iter_primary/iter_by_user are shared on the generic History and delegated by both wrappers.
• Structural duplication vs. repo: none — the accessors extend existing sibling patterns (range_primary, orders_after) rather than mirroring a new type.
• Unused derives: none (no new types).
• Primitive-obsession params: one 🔵 — iter_by_user returns the insertion seq as bare u64 (inline).
• Divergent invariant handling: none — order_book(book_id).expect("BUG…") is a genuine invariant; helpers panic consistently.
• Silent fallbacks: none — download/verify helpers panic loudly on failure; the only let _ = remove_* calls are cleanup that then panics.
• Test-only code in production: cleared — the promoted accessors are legitimate pub key-ordered read getters (parallel to existing ones), not #[cfg(test)] shims; the old #[cfg(test)] iterators on VectorMemory remain for unit-test PartialEq and are a different shape.

Test-pyramid / coverage notes

• New coverage is an integration test, which is appropriate: it needs the real mainnet snapshot + PocketIC, which unit tests can't provide, so it does not duplicate assert_replay_matches.
• Accessor set is minimal — every added pub accessor is consumed by R2/R3.
• Helper hoist is behaviour-preserving: functions/constants moved verbatim, CONTROLLER byte value unchanged, mainnet_snapshot.rs now imports them.

Verdict rationale: two 🟠 (vacuous-R3 guard, PocketIC-load duplication) plus pending CI. Not ready.

[github-actions]

canbench 🏋 (dir: canister) 51bcf0b 2026-07-22 06:14:06 UTC

✅ canister/canbench_results.yml is up to date
📦 canbench_results_benchmark.csv available in artifacts

---------------------------------------------------

Summary:
  instructions:
    status:   No significant changes 👍
    counts:   [total 16 | regressed 0 | improved 0 | new 0 | unchanged 16]
    change:   [max +28 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

  heap_increase:
    status:   No significant changes 👍
    counts:   [total 16 | regressed 0 | improved 0 | new 0 | unchanged 16]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

  stable_memory_increase:
    status:   No significant changes 👍
    counts:   [total 16 | regressed 0 | improved 0 | new 0 | unchanged 16]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

---------------------------------------------------
CSV results saved to canbench_results.csv

[Copilot]

Pull request overview

Copilot reviewed 9 out of 10 changed files in this pull request and generated 2 comments.

[gregorydemay]

🤖 🧐 VERDICT: READY — 0 blockers, 0 mediums, 1 nit; CI green. Re-review of the three first-round findings: all resolved.

Review details

First-round findings — all resolved

1. R3 vacuous-pass guard ✅ — mainnet_snapshot.rs:167-170 now asserts !state.trading_pairs().is_empty() before the order-book cross-check loop, so a replay regression that drops every pair fails instead of passing vacuously.
2. Duplication / shared load helper ✅ — the create-at-mainnet-id → add_cycles → upload → stop → load → start sequence is hoisted into load_snapshot_into_pocketic in integration_tests/src/lib.rs:657 and called by BOTH tests (mainnet_snapshot.rs:56 and :172). No near-verbatim copy remains.
3. iter_by_user typing ✅ — returns InsertionSeq (Seq<InsertionSeqMarker>) end to end through History, OrderHistory, and TradeHistory; re-exported via the order module so the public signature is nameable.

Sanity checks

• Merged single file: the upgrade test should_load_mainnet_snapshot_and_upgrade_to_current_wasm is intact, now sharing the same load helper; no regression. replay_mainnet.rs deleted, both tests in mainnet_snapshot.rs.
• Concurrency-safe download: sound. Archive is fetched to a per-call unique temp file, SHA-256-verified, then atomically renamed into the shared cached path; extraction goes to a per-call unique dir. Readers only ever see a fully-verified archive, and unique_suffix() (pid + atomic counter) prevents path collisions across parallel tests.
• Test-only-in-production concern from the prior shape is gone: UserRegistry::iter_* and TokenBalance::iter are now genuine public accessors (with docs) exercised by the tests, not #[cfg(test)] shims.

Maintainability accounting

• Duplication: none remaining (shared helper resolves the R3-load copy; the two tests now diverge only in their assertions).
• Unused derives: none (no new types; InsertionSeqMarker re-export is load-bearing for the public InsertionSeq alias).
• Primitive-obsession parameters: cleared (the u64 insertion-sequence slip is now InsertionSeq).
• Divergent invariant handling: none found.
• Silent fallbacks: none masking success — the let _ = remove_* calls are best-effort cleanup on paths that still surface the primary failure via panic!.

Coverage (R1–R4)

• R1 replay-without-panic, R2 entry-by-entry collection equality (length + per-entry via assert_entries_eq), R3 per-pair order-book depth vs PocketIC (now guarded non-empty), R4 SHA-256 pin + reject-on-mismatch — all exercised.

Nit (non-blocking)

• 🔵 The PR description still reads Spec: docs/src/development/specs/DEFI-2942-replay-mainnet-events.md, but that spec was intentionally removed — the reference now points at a nonexistent file. Drop that line from the description.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 10 changed files in this pull request and generated no new comments.

[gregorydemay]

🤖 This PR is ready for your review.

• All CI checks pass, including integration-tests (both snapshot tests run end-to-end in CI).
• Reviewer verdict: READY. All review threads resolved (reviewer findings + Copilot nits addressed with commits).
• mergeable: MERGEABLE.
• Your structural requests are in: the spec was removed, and both snapshot tests now live in integration_tests/tests/mainnet_snapshot.rs.

Left as a draft — marking ready, approval, and merge are yours.