From 66b6d355226017acfbccec1c01295a2cb16f5827 Mon Sep 17 00:00:00 2001 From: Oscar Bailey <79094698+ozgb@users.noreply.github.com> Date: Wed, 12 Aug 2026 12:10:54 +0100 Subject: [PATCH 01/13] feat(ledger): ledger 8->9 hardfork on-chain migration (#1925) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(ledger): add v8->v9 storage migration for the ledger 8->9 hardfork Port the v8->v9 state translation table from midnight-ledger PR #539 into `midnight_node_ledger::state_translation_v8_to_v9`, and wire it into the runtime as a single-block storage migration: - New host function `Ledger9Bridge::migrate_state_v8_to_v9` reads the v8 arena root (pallet-midnight `StateKey`), walks/translates the v8 `LedgerState` into the v9 shape, re-persists it, and returns the new v9 root. v8 and v9 share one storage backend, so the arena is shared. - `pallet_midnight::migrations::v2::MigrateV1ToV2` (VersionedMigration 1->2) calls the host function and re-points `StateKey`. Bumped pallet-midnight STORAGE_VERSION 1 -> 2 and added it to the runtime `Migrations` tuple. - Un-ignore the `hardfork_single_tx` e2e test and point its fork-from image at the ledger-8 release `midnightntwrk/midnight-node:1.0.1`. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> * test(ledger): add v8->v9 state translation table + smoke tests - table_is_closed / table_tags_match_types guard the translation table against tag drift on the node's rc.3 crate versions. - empty_state_translates_and_round_trips exercises an end-to-end v8->v9 translation and a v9 serialize round-trip. - Enable the helpers `can-panic` feature for ledger dev builds so the crate's `#[cfg(test)]` modules compile standalone. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> * fix(node): version-aware ledger genesis seeding for the 8->9 hardfork A ledger-9 node booting on a ledger-8 chain-spec (the hardfork fork-from case) panicked at startup: the arena seeder hardcoded the ledger-9 deserializer and rejected the v8 genesis (`expected ledger-state[v18], got ledger-state[v13]`). The genesis block runs under the old WASM, so the arena must be seeded in the genesis version. - Add `genesis_matches_this_version` (per-version tag check) to common storage. - Add `init_ledger_storage_{separate,unified}` dispatchers that pick the ledger_8 vs ledger_9 seeder by the genesis `ledger-state[vN]` tag; v8 and v9 share one backend so a v8-seeded arena is what the post-migration v9 reads. - Route `custom_parity_db` through the dispatchers. - Add a dev test proving the v8-seeded root matches the fork-from chain-spec's genesisStateKey (no `Ledger`-wrapper drift vs release 1.0.1). Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> * refactor(ledger): use serialize peek_tag for genesis version detection Replace the substring header scan in `genesis_matches_this_version` with `midnight_serialize::peek_tag`, matching the pattern used by `contract_operation_versioned_verifier_key`. Compares the peeked header tag against this version's `LedgerState::tag()` (no hardcoded version numbers). Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> * fix(ledger): restore ledger-8 construct_distribute_treasury host fn A ledger-9 node could not instantiate the ledger-8 runtime WASM (the hardfork fork-from case): the WASM imports `ext_ledger_8_bridge_construct_distribute_treasury_system_tx_version_1`, which was dropped from the current ledger-8 bridge (renamed to reserve/unlock-to-treasury for v9). Re-add it so the current node can execute the ledger-8 runtime across the 8->9 boundary. - ledger_8 builds `SystemTransaction::PayBlockRewardsToTreasury { amount }` (matching release 1.0.1); v7/v9 helpers are error stubs (only the ledger-8 bridge exposes the host fn). - Wire through the common Bridge and the Ledger8Bridge runtime interface. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> * fix(toolkit): runtime-upgrade robust to the ledger 8->9 metadata switch `runtime-upgrade` used subxt's `wait_for_finalized_success()`, which eagerly decodes the apply block's events. Across the 8->9 hardfork the client's metadata follows the code swap to the new runtime, so decoding the old runtime's `System.CodeUpdated` event fails ("Can't decode field hash ..."). Wait only for finalization (`wait_for_finalized`, no event decode) and confirm the upgrade enacted by polling `state_getRuntimeVersion` for the spec_version bump — no metadata-dependent decoding across the boundary. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> * feat(toolkit): real v8->v9 fork translation at the tx-generation boundary The toolkit built a ledger-8 transaction for the post-hardfork (ledger-9) chain, which the node rejected (Deserialization(Transaction): transaction[v9]/signature[v1] vs expected transaction[v12]/signature[v2]). Its fork-aware replay never transitioned the context ledger-8 -> ledger-9. - Move StateTranslationTable into `midnight-node-ledger-helpers` (so both the runtime migration in `ledger` and the toolkit fork can use it), adding the onchain-state deps there. - `fork_context_8_to_9` now runs the real `TypedTranslationState` translation (Db8 == Db9, one shared arena) instead of the tag-reuse `old_to_new_sp`. - Wire the ledger-8 -> ledger-9 transition into `replay_blocks` (new `fork_8_to_9_if_needed`) and relax the "not supported yet" assert. - runtime-upgrade now waits for the spec bump at the FINALIZED head (toolkit fetch reads only finalized blocks), so the post-fork fetch sees the ledger-9 blocks and the replay forks to ledger 9. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> * chore(toolkit): TEMP hardfork_debug log for replay_blocks partition Temporary diagnostic to see the l7/l8/l9 block partition + initial context version during the post-fork tx build. Remove once the fork boundary is fixed. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> * fix(toolkit): runtime-upgrade waits for the new runtime to EXECUTE, not just apply state_getRuntimeVersion(finalized) reports the *stored* code, which flips to the new runtime at the apply_authorized_upgrade block — but that block still *executes* under the old runtime (its MNSV digest, which the fetcher uses to classify ledger version, is the old spec). The first block to run the new runtime is apply+1. The prior poll returned at the apply block, so a downstream fetch (bounded by finalized height) reached the apply block (still classified ledger-8) but not apply+1, and the toolkit built a ledger-8 tx post-fork. Track the finalized height where the stored spec first bumps, then wait for the finalized height to advance past it (apply+1 finalized) before reporting success. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> * chore(toolkit): remove TEMP hardfork_debug replay_blocks log The v8->v9 fork boundary is fixed and hardfork_e2e passes; drop the diagnostic. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> * docs: add change files for the ledger 8->9 hardfork migration Assisted-by: Claude:claude-sonnet-5 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> * chore: cargo fmt Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> * chore: fix clippy errors Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> * chore: npm audit fix (toolkit-js) Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> * feat: add perf prints to track migration time elapsed Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> * feat: use ledger cost model for storage migration Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> * chore(local-environment): npm audit fix Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> * fix: no-op ledger v8->v9 migration when state is already ledger-9 The 2.0.0 runtime runs ledger-9 but shipped pallet-midnight at storage version 1 (it had no v1->v2 migration), so a network upgrading 2.0.0 -> this runtime still fires VersionedMigration<1,2> over an already-v9 StateKey. Feeding that v9 root to the v8 decode path fails on the tag mismatch and the pallet's expect() panics, bricking the upgrade block. Guard migrate_state_v8_to_v9 so it no-ops when the StateKey already references a ledger-9 state: detect it by the arena root's serialized tag (distinct between ledger-state[v13] and [v18]) and return the key unchanged with zero synthetic cost, leaving only the storage-version bump. Assisted-by: Claude:claude-fable-5 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> * docs: update change file Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> --------- Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> Co-authored-by: Squirrel (cherry picked from commit 74a91156baaf9eaf6f5fd9ba8501decd151cc841) Assisted-by: Claude:claude-opus-5 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> --- Cargo.lock | 6 + Cargo.toml | 11 +- .../ledger-8-to-9-hardfork-migration.md | 22 + .../hardfork-fork-aware-tx-generation.md | 18 + ledger/Cargo.toml | 9 + ledger/helpers/Cargo.toml | 2 + ledger/helpers/src/fork/fork_8_to_9.rs | 35 +- ledger/helpers/src/lib.rs | 5 + .../helpers/src/state_translation_v8_to_v9.rs | 710 ++++++++++++++++++ ledger/src/host_api/ledger_8.rs | 14 + ledger/src/host_api/ledger_9.rs | 26 + ledger/src/host_api/migration_8_to_9.rs | 222 ++++++ ledger/src/host_api/mod.rs | 4 + ledger/src/lib.rs | 49 ++ ledger/src/versions/common/mod.rs | 8 + ledger/src/versions/common/storage.rs | 19 + ledger/src/versions/system_tx/ledger_7.rs | 5 + ledger/src/versions/system_tx/ledger_8.rs | 8 + ledger/src/versions/system_tx/ledger_9.rs | 7 + node/src/backend/custom_parity_db.rs | 14 +- pallets/midnight/src/lib.rs | 5 +- pallets/midnight/src/migrations/mod.rs | 3 + pallets/midnight/src/migrations/v2.rs | 117 +++ runtime/src/lib.rs | 8 +- util/toolkit/src/commands/runtime_upgrade.rs | 116 ++- util/toolkit/src/tx_generator/builder/mod.rs | 41 +- util/toolkit/test-images.docker-compose.yml | 5 +- util/toolkit/tests/hardfork_e2e.rs | 1 - 28 files changed, 1444 insertions(+), 46 deletions(-) create mode 100644 changes/runtime/changed/ledger-8-to-9-hardfork-migration.md create mode 100644 changes/toolkit/changed/hardfork-fork-aware-tx-generation.md create mode 100644 ledger/helpers/src/state_translation_v8_to_v9.rs create mode 100644 ledger/src/host_api/migration_8_to_9.rs create mode 100644 pallets/midnight/src/migrations/v2.rs diff --git a/Cargo.lock b/Cargo.lock index d94267ba7..dea0ae423 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -8049,6 +8049,8 @@ dependencies = [ "midnight-onchain-runtime 2.0.1", "midnight-onchain-runtime 3.1.0", "midnight-onchain-runtime 4.0.0", + "midnight-onchain-state 3.0.0", + "midnight-onchain-state 4.0.0", "midnight-primitives-ledger", "midnight-serialize", "midnight-storage 1.1.1", @@ -8098,6 +8100,8 @@ dependencies = [ "midnight-onchain-runtime 2.0.1", "midnight-onchain-runtime 3.1.0", "midnight-onchain-runtime 4.0.0", + "midnight-onchain-state 3.0.0", + "midnight-onchain-state 4.0.0", "midnight-serialize", "midnight-storage 1.1.1", "midnight-storage 2.0.1", @@ -8705,6 +8709,7 @@ checksum = "210e601a89aee79ce2b007cf96c1a56c23baa306b0c40b9b98d12c27e18d1981" dependencies = [ "crypto", "derive-where", + "hashbrown 0.16.1", "midnight-base-crypto", "midnight-serialize", "midnight-storage-core", @@ -8726,6 +8731,7 @@ dependencies = [ "crypto", "derive-where", "fake", + "hashbrown 0.16.1", "hex", "itertools 0.14.0", "konst 0.4.3", diff --git a/Cargo.toml b/Cargo.toml index e6c6c6ede..700f473bd 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -87,8 +87,14 @@ zkir = { version = "^2.2.0", package = "midnight-zkir" } # Ledger 8 (compatible with layout-v2) # coin-structure and transient-crypto (2.x) share versions with L7 so reuse those entries. mn-ledger-8 = { version = "=8.1.0", package = "midnight-ledger" } -ledger-storage-ledger-8 = { version = "=2.0.1", package = "midnight-storage", features = ["parity-db"] } +# `state-translation` (needed by the v8->v9 storage migration) pulls in +# `public-internal-structure`, exposing `merkle_patricia_trie`/`storable`/the +# `state_translation` module used by `midnight_node_ledger::state_translation_v8_to_v9`. +ledger-storage-ledger-8 = { version = "=2.0.1", package = "midnight-storage", features = ["parity-db", "state-translation"] } onchain-runtime-ledger-8 = { version = "=3.1.0", package = "midnight-onchain-runtime" } +# Same `midnight-onchain-state` instance that `mn-ledger-8`'s `ContractState` +# resolves to (via onchain-runtime 3.1.0); used as the v8 side of the state translation table. +onchain-state-ledger-8 = { version = "=3.0.0", package = "midnight-onchain-state" } zswap-ledger-8 = { version = "=8.1.0", package = "midnight-zswap" } # Ledger 9 (compatible with layout-v2; midnight-ledger-v9 crate) @@ -97,6 +103,9 @@ zswap-ledger-8 = { version = "=8.1.0", package = "midnight-zswap" } # (midnight-zkir 2.2.0) serves all of L7/L8/L9. mn-ledger-9 = { version = "=1.0.0", package = "midnight-ledger-v9" } onchain-runtime-ledger-9 = { version = "=4.0.0", package = "midnight-onchain-runtime" } +# v9 side of the state translation table (matches `mn-ledger-9`'s `ContractState` +# via onchain-runtime 4.0.0). Patched to onchain-state-4.0.0-rc.3 by [patch.crates-io]. +onchain-state-ledger-9 = { version = "=4.0.0", package = "midnight-onchain-state" } zswap-ledger-9 = { version = "=9.0.0", package = "midnight-zswap" } coin-structure-ledger-9 = { version = "=3.0.0", package = "midnight-coin-structure" } transient-crypto-ledger-9 = { version = "=3.0.0", package = "midnight-transient-crypto" } diff --git a/changes/runtime/changed/ledger-8-to-9-hardfork-migration.md b/changes/runtime/changed/ledger-8-to-9-hardfork-migration.md new file mode 100644 index 000000000..e2c92f72c --- /dev/null +++ b/changes/runtime/changed/ledger-8-to-9-hardfork-migration.md @@ -0,0 +1,22 @@ +#node #runtime #ledger + +# On-chain ledger 8->9 hardfork state migration + +Lets a ledger-8 chain (e.g. `1.0.1`) runtime-upgrade in place to the current +ledger-9 runtime. A new host fn, `migrate_state_v8_to_v9`, runs the +`StateTranslationTable` (ported from `midnight-ledger` PR #539) to translate +the on-chain `LedgerState` from v13 to v18. It's wired in as +`pallet_midnight::migrations::v2::MigrateV1ToV2`, a `VersionedMigration<1,2,..>` +that fires once when a ledger-8 chain (pallet-midnight storage version 1) +upgrades to this runtime (storage version 2); a fresh ledger-9 genesis starts +at version 2 and skips it. The migration's weight is derived from the ledger +cost model rather than a hand-tuned estimate. + +Also includes two fixes needed to support both ledger-8 and ledger-9 chains: +version-aware genesis seeding (detected via `serialize::peek_tag` instead of +hardcoding the v9 deserializer), and restoring the ledger-8 +`construct_distribute_treasury_system_tx` host fn, which the `1.0.1` WASM +still imports. + +PR: https://github.com/midnightntwrk/midnight-node/pull/1925 +Issue: https://github.com/midnightntwrk/midnight-node/issues/1580 diff --git a/changes/toolkit/changed/hardfork-fork-aware-tx-generation.md b/changes/toolkit/changed/hardfork-fork-aware-tx-generation.md new file mode 100644 index 000000000..9fc711c0e --- /dev/null +++ b/changes/toolkit/changed/hardfork-fork-aware-tx-generation.md @@ -0,0 +1,18 @@ +#toolkit #ledger9 + +# Fork-aware transaction generation across the ledger 8->9 hardfork + +`replay_blocks` now detects the v8->v9 fork boundary and runs the same +`StateTranslationTable` translation used by the on-chain migration +(`fork_context_8_to_9` / `fork_8_to_9_if_needed`), so transactions generated +after the fork are built against the correctly-translated ledger-9 context +instead of a stale ledger-8 one. + +The `runtime-upgrade` command now waits for the new runtime to actually +*execute* at a finalized block, not just be applied/stored. The stored spec +version flips at the apply block, but that block still executes under the +old runtime, so polling only the stored spec left transaction generation +reading a block short of any ledger-9-classified block. + +PR: https://github.com/midnightntwrk/midnight-node/pull/1925 +Issue: https://github.com/midnightntwrk/midnight-node/issues/1580 diff --git a/ledger/Cargo.toml b/ledger/Cargo.toml index 4a09698cd..023ea32a9 100644 --- a/ledger/Cargo.toml +++ b/ledger/Cargo.toml @@ -18,11 +18,13 @@ zswap = { workspace = true, optional = true } mn-ledger-8 = { workspace = true, features = ["proving"], optional = true } onchain-runtime-ledger-8 = { workspace = true, optional = true } +onchain-state-ledger-8 = { workspace = true, optional = true } ledger-storage-ledger-8 = { workspace = true, optional = true } zswap-ledger-8 = { workspace = true, optional = true } mn-ledger-9 = { workspace = true, features = ["proving"], optional = true } onchain-runtime-ledger-9 = { workspace = true, optional = true } +onchain-state-ledger-9 = { workspace = true, optional = true } zswap-ledger-9 = { workspace = true, optional = true } coin-structure-ledger-9 = { workspace = true, optional = true } transient-crypto-ledger-9 = { workspace = true, optional = true } @@ -50,6 +52,11 @@ scale-info.workspace = true [dev-dependencies] midnight-node-res = { workspace = true, features = ["test", "chain-spec"] } +# The crate's own `#[cfg(test)]` modules (api::ledger, api::transaction, and the +# state-translation tests) use `extract_tx_with_context`, gated behind the +# helpers `can-panic` feature. Enable it for test builds so the tests compile +# when the crate is tested standalone. +midnight-node-ledger-helpers = { workspace = true, features = ["can-panic", "test-utils"] } [features] default = [ @@ -82,10 +89,12 @@ std = [ "zswap", "mn-ledger-8", "onchain-runtime-ledger-8", + "onchain-state-ledger-8", "ledger-storage-ledger-8", "zswap-ledger-8", "mn-ledger-9", "onchain-runtime-ledger-9", + "onchain-state-ledger-9", "zswap-ledger-9", "coin-structure-ledger-9", "transient-crypto-ledger-9", diff --git a/ledger/helpers/Cargo.toml b/ledger/helpers/Cargo.toml index 0c675d892..e46fde9c5 100644 --- a/ledger/helpers/Cargo.toml +++ b/ledger/helpers/Cargo.toml @@ -21,11 +21,13 @@ reqwest = { workspace = true } mn-ledger-8 = { workspace = true, features = ["proving"] } onchain-runtime-ledger-8 = { workspace = true } +onchain-state-ledger-8 = { workspace = true } ledger-storage-ledger-8 = { workspace = true } zswap-ledger-8 = { workspace = true } mn-ledger-9 = { workspace = true, features = ["proving", "test-utilities"] } onchain-runtime-ledger-9 = { workspace = true } +onchain-state-ledger-9 = { workspace = true } zswap-ledger-9 = { workspace = true } coin-structure-ledger-9 = { workspace = true } transient-crypto-ledger-9 = { workspace = true } diff --git a/ledger/helpers/src/fork/fork_8_to_9.rs b/ledger/helpers/src/fork/fork_8_to_9.rs index 918cda2e9..fac03ec49 100644 --- a/ledger/helpers/src/fork/fork_8_to_9.rs +++ b/ledger/helpers/src/fork/fork_8_to_9.rs @@ -2,6 +2,10 @@ use std::collections::HashMap; use tokio::sync::Mutex as MutexTokio; +use crate::state_translation_v8_to_v9::StateTranslationTable; +use base_crypto::cost_model::CostDuration; +use ledger_storage_ledger_8::state_translation::TypedTranslationState; + type Db8 = crate::ledger_8::DefaultDB; type Db9 = crate::ledger_9::DefaultDB; @@ -67,8 +71,35 @@ pub fn fork_context_8_to_9( context8: LedgerContext8, ) -> Result, std::io::Error> { let ledger_state_8 = context8.ledger_state.lock().expect("failed to lock ledger state"); - let ledger_state: crate::ledger_9::Sp, Db8> = - old_to_new_sp(ledger_state_8.clone())?; + // Real v8->v9 state translation (NOT `old_to_new_sp`): the LedgerState tag + // changed v13->v18 and its shape changed, so a bare arena-key reuse would + // produce a v9 root the ledger-9 machinery can't read. Walk the v8 state + // through the same `StateTranslationTable` the on-chain migration uses. Db8 + // == Db9 (both `ledger_storage_ledger_8::DefaultDB`), so source and target + // share one arena and a single-`D` `TypedTranslationState` applies. + let ledger_state: crate::ledger_9::Sp, Db8> = { + let mut tl = TypedTranslationState::< + mn_ledger_8::structure::LedgerState, + mn_ledger_9::structure::LedgerState, + StateTranslationTable, + Db8, + >::start(ledger_state_8.clone())?; + // Single-shot: a generous per-step budget drains the whole state in a + // couple of iterations; the step cap is only a runaway backstop. + // 1_000_000_000_000 pico-seconds == 1 second + let budget = CostDuration::from_picoseconds(1_000_000_000_000); + let mut steps = 0usize; + loop { + steps += 1; + if steps > 100_000 { + return Err(std::io::Error::other("v8->v9 state translation did not converge")); + } + tl = tl.run(budget)?; + if let Some(result) = tl.result()? { + break result; + } + } + }; let mut wallets = HashMap::new(); for (k, v) in context8.wallets.lock().expect("failed to lock wallets").iter() { diff --git a/ledger/helpers/src/lib.rs b/ledger/helpers/src/lib.rs index 06494e10f..e2f67bdf2 100644 --- a/ledger/helpers/src/lib.rs +++ b/ledger/helpers/src/lib.rs @@ -16,6 +16,11 @@ mod utils; pub use utils::find_dependency_version; pub mod extract_tx_with_context; +/// v8 -> v9 ledger state translation table (ported from midnight-ledger PR #539). +/// Consumed by the runtime storage migration (via the `ledger` crate) and by the +/// toolkit fork boundary (`fork::fork_8_to_9`). +pub mod state_translation_v8_to_v9; + /// Strategy for ordering candidate coins/UTXOs during input selection. /// /// Defined at the crate root (not inside the version-specific `common` module) so that diff --git a/ledger/helpers/src/state_translation_v8_to_v9.rs b/ledger/helpers/src/state_translation_v8_to_v9.rs new file mode 100644 index 000000000..4106b7819 --- /dev/null +++ b/ledger/helpers/src/state_translation_v8_to_v9.rs @@ -0,0 +1,710 @@ +// This file is part of midnight-node. +// Copyright (C) 2025-2026 Midnight Foundation +// SPDX-License-Identifier: Apache-2.0 +// Licensed under the Apache License, Version 2.0 (the "License"); +// You may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! State translation from ledger v8 to ledger v9. +//! +//! Ported from `midnight-ledger` PR #539 (`v8-to-v9-state-translation`). The +//! only changes from the upstream crate are the import aliases below, which map +//! the translation's `ledger_v8` / `ledger_v9` / `onchain_state_v8` / +//! `onchain_state_v9` / `storage` / `serialize` crate names onto this +//! workspace's package aliases. The [`StateTranslationTable`] is consumed by the +//! v8->v9 storage migration ([`crate::host_api::migration_8_to_9`]). +//! +//! ## State shape differences (only stored types listed) +//! +//! | type | v8 tag | v9 tag | change | +//! | ---------------------------- | ------------------------------------ | ------------------------------------ | ------ | +//! | LedgerState | `ledger-state[v13]` | `ledger-state[v18]` | `bridge_receiving` map gains `NightAnn` | +//! | LedgerParameters | `ledger-parameters[v5]` | `ledger-parameters[v8]` | adds `min_block_price`; `TransactionLimits` adds `max_contract_metadata_size`; `TransactionCostModel` drops `parallelism_factor`, adds `validation`/`guaranteed`/`fallible` factors | +//! | ContractState | `contract-state[v6]` | `contract-state[v8]` | reflows `ContractOperation` + `ContractMaintenanceAuthority` changes | +//! | ContractOperation | `contract-operation[v4]` | `contract-operation[v6]` | single `v2` key -> `{ v2, v3, ir }`; v8 key maps to `v2`, new `v3`/`ir` empty | +//! | ContractMaintenanceAuthority | `contract-maintenance-authority[v1]` | `contract-maintenance-authority[v2]` | `committee: Vec` -> `Vec` (Schnorr/ECDSA sum) | +//! +//! Everything else (zswap, utxo, dust, replay_protection, treasury, +//! unclaimed_block_rewards) is tag-stable and passes through `recast`. + +// Map the upstream translation crate names onto the node workspace's package +// aliases. `mn-ledger-8`/`mn-ledger-9` are the two `midnight-ledger` majors; +// `onchain-state-ledger-8`/`-9` are the exact `midnight-onchain-state` instances +// that each ledger's `ContractState` resolves to; `ledger-storage-ledger-8` +// (midnight-storage 2.0.1, `state-translation` feature) backs both. +use ledger_storage_ledger_8 as storage; +use midnight_serialize as serialize; +use mn_ledger_8 as ledger_v8; +use mn_ledger_9 as ledger_v9; +use onchain_state_ledger_8 as onchain_state_v8; +use onchain_state_ledger_9 as onchain_state_v9; + +use base_crypto::cost_model::CostDuration; +use serialize::Tagged; +use std::ops::Deref; +use std::{any::Any, borrow::Cow, io, marker::PhantomData}; +use storage::{ + Storable, + arena::Sp, + db::DB, + merkle_patricia_trie::{self, Annotation, MerklePatriciaTrie}, + state_translation::*, + storable::SizeAnn, + storage::{HashMap, Map, default_storage}, +}; + +// ---------- Generic helpers (copied from the v6->v7 reference) ---------- + +/// Recast a stored object from one type to another, requiring matching tags. +/// Used for subtrees whose tag is unchanged between v8 and v9. +fn recast + Tagged, B: Storable + Tagged, D: DB>( + a: &Sp, +) -> io::Result> { + if A::tag() != B::tag() { + return io::Result::Err(io::Error::other("tags do not match")); + } + default_storage::().get_lazy(&a.as_child().into()) +} + +/// Generic MPT translation: walks the trie, translating each entry via the +/// table-registered translation for `A->B`, and recomputes annotations under +/// `AnnB` from the new values. +struct MptTl(PhantomData<(A, B, AnnA, AnnB)>); + +impl< + A: Storable + Tagged, + B: Storable + Tagged, + AnnA: Annotation + Storable + Tagged, + AnnB: Annotation + Storable + Tagged, + D: DB, +> DirectTranslation, MerklePatriciaTrie, D> + for MptTl +{ + fn required_translations() -> Vec { + vec![TranslationId( + merkle_patricia_trie::Node::::tag(), + merkle_patricia_trie::Node::::tag(), + )] + } + fn child_translations( + source: &MerklePatriciaTrie, + ) -> Vec<(TranslationId, Sp)> { + let tlids = , _, D>>::required_translations(); + vec![(tlids[0].clone(), source.0.upcast())] + } + fn finalize( + source: &MerklePatriciaTrie, + _limit: &mut CostDuration, + cache: &TranslationCache, + ) -> io::Result>> { + let tls = Self::child_translations(source); + Ok(Some(MerklePatriciaTrie(try_resopt!(cache.resolve(&tls[0].0, tls[0].1.as_child()))))) + } +} + +impl< + A: Storable + Tagged, + B: Storable + Tagged, + AnnA: Storable + Tagged + Annotation, + AnnB: Storable + Tagged + Annotation, + D: DB, +> + DirectTranslation< + merkle_patricia_trie::Node, + merkle_patricia_trie::Node, + D, + > for MptTl +{ + fn required_translations() -> Vec { + let entry_tl = TranslationId(A::tag(), B::tag()); + let self_tl = TranslationId( + merkle_patricia_trie::Node::::tag(), + merkle_patricia_trie::Node::::tag(), + ); + vec![entry_tl, self_tl] + } + fn child_translations( + source: &merkle_patricia_trie::Node, + ) -> Vec<(TranslationId, Sp)> { + let tls = , _, D>>::required_translations(); + let entry_tl = tls[0].clone(); + let self_tl = tls[1].clone(); + match source { + merkle_patricia_trie::Node::Empty => vec![], + merkle_patricia_trie::Node::Branch { children, .. } => { + children.iter().map(|child| (self_tl.clone(), child.upcast())).collect() + }, + merkle_patricia_trie::Node::Extension { child, .. } => { + vec![(self_tl, child.upcast())] + }, + merkle_patricia_trie::Node::MidBranchLeaf { value, child, .. } => { + vec![(entry_tl, value.upcast()), (self_tl, child.upcast())] + }, + merkle_patricia_trie::Node::Leaf { value, .. } => vec![(entry_tl, value.upcast())], + } + } + fn finalize( + source: &merkle_patricia_trie::Node, + _limit: &mut CostDuration, + cache: &TranslationCache, + ) -> io::Result>> { + let tls = Self::child_translations(source); + Ok(Some(match source { + merkle_patricia_trie::Node::Empty => merkle_patricia_trie::Node::Empty, + merkle_patricia_trie::Node::Branch { .. } => { + let mut new_children = + core::array::from_fn(|_| Sp::new(merkle_patricia_trie::Node::Empty)); + for (child, new_child) in tls.iter().zip(new_children.iter_mut()) { + *new_child = try_resopt!(cache.resolve(&child.0, child.1.as_child())); + } + let ann = new_children.iter().fold(AnnB::empty(), |acc, x| { + acc.append(&merkle_patricia_trie::Node::::ann(x)) + }); + merkle_patricia_trie::Node::Branch { ann, children: Box::new(new_children) } + }, + merkle_patricia_trie::Node::Extension { compressed_path, .. } => { + let child: Sp, D> = + try_resopt!(cache.resolve(&tls[0].0, tls[0].1.as_child())); + let ann = merkle_patricia_trie::Node::::ann(&child); + merkle_patricia_trie::Node::Extension { + ann, + compressed_path: compressed_path.clone(), + child, + } + }, + merkle_patricia_trie::Node::Leaf { .. } => { + let value = try_resopt!(cache.resolve(&tls[0].0, tls[0].1.as_child())); + let ann = AnnB::from_value(&value); + merkle_patricia_trie::Node::Leaf { ann, value } + }, + merkle_patricia_trie::Node::MidBranchLeaf { .. } => { + let value = try_resopt!(cache.resolve(&tls[0].0, tls[0].1.as_child())); + let child: Sp, D> = + try_resopt!(cache.resolve(&tls[1].0, tls[1].1.as_child())); + let ann = AnnB::from_value(&value) + .append(&merkle_patricia_trie::Node::::ann(&child)); + merkle_patricia_trie::Node::MidBranchLeaf { ann, value, child } + }, + })) + } +} + +/// Identity translation for a type whose serialization is unchanged across +/// versions. Needed when an MPT's entries are tag-stable but its annotation +/// changes (e.g. `bridge_receiving`). +struct IdentityTl(PhantomData); + +impl + Clone, D: DB> DirectTranslation for IdentityTl { + fn required_translations() -> Vec { + Vec::new() + } + fn child_translations(_: &T) -> Vec<(TranslationId, Sp)> { + Vec::new() + } + fn finalize( + source: &T, + _limit: &mut CostDuration, + _cache: &TranslationCache, + ) -> io::Result> { + Ok(Some(source.clone())) + } +} + +// ---------- Translation IDs (shorthand) ---------- + +struct Ids; + +impl Ids { + fn contract_mpt() -> TranslationId { + TranslationId( + MerklePatriciaTrie::< + onchain_state_v8::state::ContractState, + D, + ledger_v8::annotation::NightAnn, + >::tag(), + MerklePatriciaTrie::< + onchain_state_v9::state::ContractState, + D, + ledger_v9::annotation::NightAnn, + >::tag(), + ) + } + + fn bridge_receiving_mpt() -> TranslationId { + TranslationId( + MerklePatriciaTrie::::tag(), + MerklePatriciaTrie::::tag(), + ) + } + + fn parameters() -> TranslationId { + TranslationId( + ledger_v8::structure::LedgerParameters::tag(), + ledger_v9::structure::LedgerParameters::tag(), + ) + } +} + +// ---------- Top-level: LedgerState v8 -> v9 ---------- + +struct LedgerStateTl; + +impl + DirectTranslation, ledger_v9::structure::LedgerState, D> + for LedgerStateTl +{ + fn required_translations() -> Vec { + vec![Ids::parameters(), Ids::bridge_receiving_mpt::(), Ids::contract_mpt::()] + } + + fn child_translations( + source: &ledger_v8::structure::LedgerState, + ) -> Vec<(TranslationId, Sp)> { + vec![ + (Ids::parameters(), source.parameters.upcast()), + (Ids::bridge_receiving_mpt::(), source.bridge_receiving.mpt.upcast()), + (Ids::contract_mpt::(), source.contract.mpt.upcast()), + ] + } + + fn finalize( + source: &ledger_v8::structure::LedgerState, + _limit: &mut CostDuration, + cache: &TranslationCache, + ) -> io::Result>> { + let Some(parameters) = cache.lookup(&Ids::parameters(), source.parameters.as_child()) + else { + return Ok(None); + }; + let Some(bridge_recv_mpt) = + cache.lookup(&Ids::bridge_receiving_mpt::(), source.bridge_receiving.mpt.as_child()) + else { + return Ok(None); + }; + let Some(contract_mpt) = + cache.lookup(&Ids::contract_mpt::(), source.contract.mpt.as_child()) + else { + return Ok(None); + }; + + Ok(Some(ledger_v9::structure::LedgerState { + network_id: source.network_id.clone(), + parameters: parameters.force_downcast(), + locked_pool: source.locked_pool, + bridge_receiving: Map { mpt: bridge_recv_mpt.force_downcast(), key_type: PhantomData }, + reserve_pool: source.reserve_pool, + block_reward_pool: source.block_reward_pool, + unclaimed_block_rewards: Map { + mpt: recast(&source.unclaimed_block_rewards.mpt)?, + key_type: PhantomData, + }, + treasury: Map { mpt: recast(&source.treasury.mpt)?, key_type: PhantomData }, + zswap: recast(&source.zswap)?, + contract: Map { mpt: contract_mpt.force_downcast(), key_type: PhantomData }, + utxo: recast(&source.utxo)?, + replay_protection: recast(&source.replay_protection)?, + dust: recast(&source.dust)?, + })) + } +} + +// ---------- LedgerParameters v8 -> v9 ---------- + +struct LedgerParametersTl; + +impl + DirectTranslation< + ledger_v8::structure::LedgerParameters, + ledger_v9::structure::LedgerParameters, + D, + > for LedgerParametersTl +{ + fn required_translations() -> Vec { + Vec::new() + } + fn child_translations( + _: &ledger_v8::structure::LedgerParameters, + ) -> Vec<(TranslationId, Sp)> { + Vec::new() + } + fn finalize( + source: &ledger_v8::structure::LedgerParameters, + _limit: &mut CostDuration, + _cache: &TranslationCache, + ) -> io::Result> { + // Base-crypto-backed fields (Duration, FixedPoint, primitives) are + // assignable directly because `midnight-base-crypto` is unified across + // v8 and v9 by workspace patches. Composite types defined in `ledger` + // (TransactionCostModel, dust parameters, etc.) are tag-stable but not + // identical types, so we go through the (de)serializer. + // + // `TransactionLimits` is the exception: v9 bumped it to + // `transaction-limits[v3]` by adding `max_contract_metadata_size`, so + // it is no longer tag-stable and is rebuilt field-by-field (its other + // fields are unified base-crypto types). + Ok(Some(ledger_v9::structure::LedgerParameters { + // `TransactionCostModel` bumped `transaction-cost-model[v4]`->`[v5]`: + // v9 drops `parallelism_factor` and adds three `FixedPoint` factors. + // The two surviving fields are tag-stable and recast through; the new + // factors get the v9 INITIAL_PARAMETERS defaults. + cost_model: ledger_v9::structure::TransactionCostModel { + runtime_cost_model: recast_base(&source.cost_model.runtime_cost_model)?, + baseline_cost: recast_base(&source.cost_model.baseline_cost)?, + // NEW IN v9 — placeholder; the production value should match the + // value chosen for the hardfork. + validation_factor: ledger_v9::structure::INITIAL_PARAMETERS + .cost_model + .validation_factor, + guaranteed_factor: ledger_v9::structure::INITIAL_PARAMETERS + .cost_model + .guaranteed_factor, + fallible_factor: ledger_v9::structure::INITIAL_PARAMETERS + .cost_model + .fallible_factor, + }, + limits: ledger_v9::structure::TransactionLimits { + transaction_byte_limit: source.limits.transaction_byte_limit, + time_to_dismiss_per_byte: source.limits.time_to_dismiss_per_byte, + min_time_to_dismiss: source.limits.min_time_to_dismiss, + block_limits: source.limits.block_limits, + block_withdrawal_minimum_multiple: source.limits.block_withdrawal_minimum_multiple, + // NEW IN v9 — placeholder; the production value should match + // the value chosen for the hardfork. + max_contract_metadata_size: ledger_v9::structure::INITIAL_PARAMETERS + .limits + .max_contract_metadata_size, + }, + dust: recast_base(&source.dust)?, + fee_prices: recast_base(&source.fee_prices)?, + global_ttl: source.global_ttl, + cost_dimension_min_ratio: source.cost_dimension_min_ratio, + price_adjustment_a_parameter: source.price_adjustment_a_parameter, + cardano_to_midnight_bridge_fee_basis_points: source + .cardano_to_midnight_bridge_fee_basis_points, + c_to_m_bridge_min_amount: source.c_to_m_bridge_min_amount, + // NEW IN v9 — placeholder; the production value should match the + // value chosen for the hardfork. + min_block_price: ledger_v9::structure::INITIAL_PARAMETERS.min_block_price, + })) + } +} + +/// Recast for tag-stable base types passed by value (cost model, limits, etc.). +/// Not the same as `recast` above which only works for `Sp`. +fn recast_base( + a: &A, +) -> io::Result { + if A::tag() != B::tag() { + return Err(io::Error::other("tags do not match")); + } + let mut buf = Vec::new(); + a.serialize(&mut buf)?; + B::deserialize(&mut &buf[..], 0) +} + +// ---------- ContractOperation v8 -> v9 ---------- + +/// Translate a single contract operation. v9 grew `ContractOperation` from a +/// single `v2` verifier key (`contract-operation[v4]`) to `{ v2, v3, ir }` +/// (`contract-operation[v6]`). v8's only key is a zk-stdlib-v1 key +/// (`verifier-key[v6]`), which v9 keeps in its `v2` slot: that slot is backed by +/// the same `transient-crypto` 2.x crate (`transient_crypto_old`), so it is the +/// identical type and assigns directly. The new zk-stdlib-v2 `v3` key +/// (`verifier-key[v7]`, transient-crypto 3.x) and the `ir` slot have no v8 +/// equivalent and stay empty — v9 keys are *not* synthesized from v8 keys. +/// (Note `ContractOperation::new(vk, ir)` sets `v3`, not `v2`, so the struct is +/// built field-wise here.) +fn translate_contract_operation( + source: &onchain_state_v8::state::ContractOperation, +) -> onchain_state_v9::state::ContractOperation { + // `ContractOperation` is `#[non_exhaustive]`; `new` seeds `v3`/`ir`, and the + // v8 key goes into the `v2` slot field-wise. + let mut op = onchain_state_v9::state::ContractOperation::new(None, None); + op.v2 = source.v2.clone(); + op +} + +// ---------- ContractState v8 -> v9 ---------- + +struct ContractStateTl; + +impl + DirectTranslation< + onchain_state_v8::state::ContractState, + onchain_state_v9::state::ContractState, + D, + > for ContractStateTl +{ + fn required_translations() -> Vec { + Vec::new() + } + fn child_translations( + _: &onchain_state_v8::state::ContractState, + ) -> Vec<(TranslationId, Sp)> { + Vec::new() + } + fn finalize( + source: &onchain_state_v8::state::ContractState, + _limit: &mut CostDuration, + _cache: &TranslationCache, + ) -> io::Result>> { + // `operations` entries (ContractOperation) changed shape, so the map + // is rebuilt entry-by-entry. The translation machinery can't walk these + // base-storable leaves nested under a contract, but a contract's + // operation set is small, so an in-place rebuild is fine. ChargedState + // and the balance map (keyed u128) are tag-stable and recast through. + let mut operations = HashMap::new(); + for entry in source.operations.iter() { + let (key, op) = &*entry; + let key_v9: onchain_state_v9::state::EntryPointBuf = key[..].into(); + operations = operations.insert(key_v9, translate_contract_operation(op)); + } + let committee_v9 = source + .maintenance_authority + .committee + .iter() + .map(|vk| onchain_state_v9::state::ContractMaintenanceVerifyingKey::Schnorr(vk.clone())) + .collect(); + let maintenance_authority = onchain_state_v9::state::ContractMaintenanceAuthority { + committee: committee_v9, + threshold: source.maintenance_authority.threshold, + counter: source.maintenance_authority.counter, + }; + Ok(Some(onchain_state_v9::state::ContractState:: { + data: recast::< + onchain_state_v8::state::ChargedState, + onchain_state_v9::state::ChargedState, + D, + >(&Sp::new(source.data.clone()))? + .deref() + .clone(), + operations, + maintenance_authority, + balance: HashMap(Map { mpt: recast(&source.balance.0.mpt)?, key_type: PhantomData }), + })) + } +} + +// ---------- Translation table ---------- + +pub struct StateTranslationTable; + +impl TranslationTable for StateTranslationTable { + const TABLE: &[(TranslationId, &dyn TypelessTranslation)] = &[ + // Top-level + ( + TranslationId(Cow::Borrowed("ledger-state[v13]"), Cow::Borrowed("ledger-state[v18]")), + &DirectSpTranslation::<_, _, LedgerStateTl, _>(PhantomData), + ), + // LedgerParameters + ( + TranslationId( + Cow::Borrowed("ledger-parameters[v5]"), + Cow::Borrowed("ledger-parameters[v8]"), + ), + &DirectSpTranslation::<_, _, LedgerParametersTl, _>(PhantomData), + ), + // ContractState + ( + TranslationId(Cow::Borrowed("contract-state[v6]"), Cow::Borrowed("contract-state[v8]")), + &DirectSpTranslation::<_, _, ContractStateTl, _>(PhantomData), + ), + // `contract` MPT in LedgerState — entries are ContractState + ( + TranslationId( + Cow::Borrowed("mpt(contract-state[v6],night-annotation)"), + Cow::Borrowed("mpt(contract-state[v8],night-annotation)"), + ), + &DirectSpTranslation::< + MerklePatriciaTrie< + onchain_state_v8::state::ContractState, + D, + ledger_v8::annotation::NightAnn, + >, + MerklePatriciaTrie< + onchain_state_v9::state::ContractState, + D, + ledger_v9::annotation::NightAnn, + >, + MptTl< + onchain_state_v8::state::ContractState, + onchain_state_v9::state::ContractState, + ledger_v8::annotation::NightAnn, + ledger_v9::annotation::NightAnn, + >, + _, + >(PhantomData), + ), + ( + TranslationId( + Cow::Borrowed("mpt-node(contract-state[v6],night-annotation)"), + Cow::Borrowed("mpt-node(contract-state[v8],night-annotation)"), + ), + &DirectSpTranslation::< + merkle_patricia_trie::Node< + onchain_state_v8::state::ContractState, + D, + ledger_v8::annotation::NightAnn, + >, + merkle_patricia_trie::Node< + onchain_state_v9::state::ContractState, + D, + ledger_v9::annotation::NightAnn, + >, + MptTl< + onchain_state_v8::state::ContractState, + onchain_state_v9::state::ContractState, + ledger_v8::annotation::NightAnn, + ledger_v9::annotation::NightAnn, + >, + _, + >(PhantomData), + ), + // `bridge_receiving` MPT — entries unchanged (u128), annotation changes + // from SizeAnn to NightAnn. Needs an identity entry translation and an + // MptTl that re-annotates. + ( + TranslationId(Cow::Borrowed("u128"), Cow::Borrowed("u128")), + &DirectSpTranslation::, _>(PhantomData), + ), + ( + TranslationId( + Cow::Borrowed("mpt(u128,size-annotation)"), + Cow::Borrowed("mpt(u128,night-annotation)"), + ), + &DirectSpTranslation::< + MerklePatriciaTrie, + MerklePatriciaTrie, + MptTl, + _, + >(PhantomData), + ), + ( + TranslationId( + Cow::Borrowed("mpt-node(u128,size-annotation)"), + Cow::Borrowed("mpt-node(u128,night-annotation)"), + ), + &DirectSpTranslation::< + merkle_patricia_trie::Node, + merkle_patricia_trie::Node, + MptTl, + _, + >(PhantomData), + ), + ]; +} + +#[cfg(test)] +mod tests { + use super::*; + use storage::db::InMemoryDB; + + fn translate_to_completion( + v8: ledger_v8::structure::LedgerState, + ) -> ledger_v9::structure::LedgerState { + let tl_state = TypedTranslationState::< + ledger_v8::structure::LedgerState, + ledger_v9::structure::LedgerState, + StateTranslationTable, + InMemoryDB, + >::start(Sp::new(v8)) + .expect("Failed to start translation"); + + let cost = CostDuration::from_picoseconds(1_000_000_000_000); + let finished = tl_state.run(cost).expect("Translation failed"); + + finished + .result() + .expect("Failed to get result") + .expect("Translation did not complete") + .deref() + .clone() + } + + /// Every `TranslationId` a table entry requires must itself be in the table, + /// or translation errors at runtime the first time the entry is needed. + #[test] + fn table_is_closed() { + >::assert_closure(); + } + + /// The `TABLE` hardcodes tag string literals. If a tag on either the v8 or v9 + /// side drifts (e.g. an rc bump changes a `#[tag]`), the literal no longer + /// matches what `T::tag()` produces and the migration silently mis-dispatches. + /// Rebuild every expected ID from the node's actual crate types and compare. + #[test] + fn table_tags_match_types() { + use storage::merkle_patricia_trie::{MerklePatriciaTrie, Node}; + use storage::storable::SizeAnn; + + type V8Ann = ledger_v8::annotation::NightAnn; + type V9Ann = ledger_v9::annotation::NightAnn; + type V8Contract = onchain_state_v8::state::ContractState; + type V9Contract = onchain_state_v9::state::ContractState; + + let expected: Vec<(Cow<'static, str>, Cow<'static, str>)> = vec![ + ( + ledger_v8::structure::LedgerState::::tag(), + ledger_v9::structure::LedgerState::::tag(), + ), + ( + ledger_v8::structure::LedgerParameters::tag(), + ledger_v9::structure::LedgerParameters::tag(), + ), + (V8Contract::tag(), V9Contract::tag()), + ( + MerklePatriciaTrie::::tag(), + MerklePatriciaTrie::::tag(), + ), + ( + Node::::tag(), + Node::::tag(), + ), + (u128::tag(), u128::tag()), + ( + MerklePatriciaTrie::::tag(), + MerklePatriciaTrie::::tag(), + ), + (Node::::tag(), Node::::tag()), + ]; + + let actual: Vec<_> = >::TABLE + .iter() + .map(|(id, _)| (id.0.clone(), id.1.clone())) + .collect(); + + assert_eq!(actual, expected); + } + + /// End-to-end smoke test: a default v8 `LedgerState` translates to v9, + /// preserving the tag-stable pools and picking up the new v9 default + /// `min_block_price`, and survives a v9 serialize round-trip. + #[test] + fn empty_state_translates_and_round_trips() { + let v8 = ledger_v8::structure::LedgerState::::new("test-network"); + let v9 = translate_to_completion(v8.clone()); + + assert_eq!(v9.network_id, v8.network_id); + assert_eq!(v9.reserve_pool, v8.reserve_pool); + assert_eq!(v9.locked_pool, v8.locked_pool); + assert_eq!(v9.block_reward_pool, v8.block_reward_pool); + assert_eq!( + v9.parameters.min_block_price, + ledger_v9::structure::INITIAL_PARAMETERS.min_block_price, + ); + + let mut buf = Vec::new(); + serialize::tagged_serialize(&v9, &mut buf).expect("v9 serialize"); + let v9_rt: ledger_v9::structure::LedgerState = + serialize::tagged_deserialize(&mut &buf[..]).expect("v9 deserialize"); + assert_eq!(v9_rt.network_id, v9.network_id); + } +} diff --git a/ledger/src/host_api/ledger_8.rs b/ledger/src/host_api/ledger_8.rs index 3222f875e..de03ac54e 100644 --- a/ledger/src/host_api/ledger_8.rs +++ b/ledger/src/host_api/ledger_8.rs @@ -429,6 +429,20 @@ pub trait Ledger8Bridge { } } + /// The ledger-8 runtime imports this to pay block rewards to the treasury. + /// Retained (removed for v9) so the current node can execute the ledger-8 + /// WASM across the 8->9 hardfork boundary. + fn construct_distribute_treasury_system_tx( + &mut self, + amount: PassFatPointerAndDecode, + ) -> AllocateAndReturnByCodec, LedgerApiError>> { + if is_unified(*self) { + Bridge::::construct_distribute_treasury_system_tx(amount) + } else { + Bridge::::construct_distribute_treasury_system_tx(amount) + } + } + /// Ensures the correct ledger storage is initialized for this runtime version. /// Handles rollback: if new version's storage is initialized but we need this version's storage, /// drops new version's storage and initializes normal storage. diff --git a/ledger/src/host_api/ledger_9.rs b/ledger/src/host_api/ledger_9.rs index 53f234c4c..2e1ca7d61 100644 --- a/ledger/src/host_api/ledger_9.rs +++ b/ledger/src/host_api/ledger_9.rs @@ -469,6 +469,32 @@ pub trait Ledger9Bridge { true } + /// Translate the ledger state from ledger-v8 format to ledger-v9 format. + /// + /// Called by `pallet_midnight`'s v8->v9 storage migration during the runtime + /// upgrade that crosses into ledger-9. `state_key` is the pallet's `StateKey` + /// (a v8 arena root); returns the new v9 arena root to store back, together + /// with the synthetic cost (picoseconds) the translation consumed against + /// the ledger's cost model, for the pallet to charge as this migration's + /// weight. + fn migrate_state_v8_to_v9( + &mut self, + state_key: PassFatPointerAndRead<&[u8]>, + ) -> AllocateAndReturnByCodec, u64), LedgerApiError>> { + // Ensure the ledger arena is initialized before translating. The migration + // runs in the Executive migrations tuple, before pallet_midnight's + // on_initialize/on_runtime_upgrade have (re)initialized storage this block. + // `set_default_storage` is idempotent — a no-op if the pre-fork ledger-8 + // blocks already set it (v8 and v9 share the same storage backend). + if is_unified(*self) { + Bridge::::set_default_storage(*self); + crate::host_api::migration_8_to_9::migrate_state_v8_to_v9::(state_key) + } else { + Bridge::::set_default_storage(*self); + crate::host_api::migration_8_to_9::migrate_state_v8_to_v9::(state_key) + } + } + /// Initialize a process-wide temporary ledger ParityDb seeded with the /// undeployed-network genesis state. /// diff --git a/ledger/src/host_api/migration_8_to_9.rs b/ledger/src/host_api/migration_8_to_9.rs new file mode 100644 index 000000000..6fe05c873 --- /dev/null +++ b/ledger/src/host_api/migration_8_to_9.rs @@ -0,0 +1,222 @@ +// This file is part of midnight-node. +// Copyright (C) 2025-2026 Midnight Foundation +// SPDX-License-Identifier: Apache-2.0 +// Licensed under the Apache License, Version 2.0 (the "License"); +// You may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Host-side v8 -> v9 ledger state translation, driven by +//! [`crate::state_translation_v8_to_v9::StateTranslationTable`]. +//! +//! The on-chain pallet stores only the arena root of the ledger state +//! (`pallet_midnight::StateKey`, a `tagged_serialize`d `TypedArenaKey`); +//! the `LedgerState` itself lives in the process-global ledger arena (parity-db). +//! When the runtime upgrades from a ledger-8 runtime (spec < 2_000_000) to a +//! ledger-9 runtime (spec >= 2_000_000), the on-chain migration +//! (`pallet_midnight::migrations`) calls [`Ledger9Bridge::migrate_state_v8_to_v9`] +//! which lands here: it reads the v8 arena root, walks the v8 `LedgerState` +//! translating it into a v9 `LedgerState`, re-persists it, and returns the new v9 +//! arena root for the pallet to store back into `StateKey`. +//! +//! v8 and v9 share one storage crate (`ledger-storage-ledger-8`, +//! midnight-storage 2.0.1) and hence one arena, so the translation reads and +//! writes the same parity-db instance the pre-fork ledger-8 blocks populated. + +use crate::ledger_8::api::Ledger as Ledger8; +use crate::ledger_9::api::Ledger as Ledger9; +use crate::ledger_9::types::{DeserializationError, LedgerApiError, SerializationError}; +use midnight_node_ledger_helpers::state_translation_v8_to_v9::StateTranslationTable; + +use base_crypto::cost_model::CostDuration; +use ledger_storage_ledger_8 as storage; +use midnight_serialize::{tagged_deserialize, tagged_serialize}; +use storage::{ + arena::{Sp, TypedArenaKey}, + db::DB, + state_translation::TypedTranslationState, + storage::default_storage, +}; + +type LedgerState8 = mn_ledger_8::structure::LedgerState; +type LedgerState9 = mn_ledger_9::structure::LedgerState; + +const LOG_TARGET: &str = "midnight::ledger::migration_8_to_9"; + +/// Picoseconds granted to each `TypedTranslationState::run` step. The migration +/// is single-block (it must complete within one host call), so we loop over +/// `run` with a per-step budget until the translation reports a result. +/// +/// This budget also doubles as the quantization granularity of the synthetic +/// cost reported back to the pallet (see `consumed_cost_ps` below): `run` +/// doesn't return unused budget, so every completed step is charged in full. +/// 10ms keeps that over-approximation small relative to real translation +/// costs while still comfortably draining dev/undeployed-sized state in a +/// handful of iterations; the loop cap is a runaway backstop only. +const RUN_BUDGET_PS: u64 = 10_000_000_000; +const MAX_STEPS: usize = 100_000; + +/// Translate the ledger state referenced by a v8 arena root (`state_key_v8`, +/// the pallet's `StateKey` bytes) into a v9 ledger state, persist it, and +/// return the new v9 arena root (to store back into `StateKey`) together with +/// the synthetic cost, in picoseconds, the translation consumed against the +/// ledger's deterministic cost model — for the pallet to charge as this +/// migration's weight. +/// +/// If `state_key_v8` already references a ledger-9 state this is a no-op: it +/// returns the key unchanged with zero cost (see the idempotency guard below). +pub fn migrate_state_v8_to_v9( + state_key_v8: &[u8], +) -> Result<(Vec, u64), LedgerApiError> { + let t_total = std::time::Instant::now(); + + // 0. Idempotency guard: no-op if the state is already ledger-9. + // + // The pallet gates this behind `VersionedMigration<1, 2>`, but the on-chain + // pallet-midnight storage version is not a faithful proxy for the ledger + // version. The 2.0.0 runtime (spec 2_000_000) already runs ledger-9 yet + // shipped pallet-midnight at storage version 1 (it had no v1->v2 migration), + // so a network upgrading 2.0.0 -> this runtime still triggers this migration + // even though its `StateKey` already points at a v9 arena root. Feeding that + // v9 root to the v8 decode below would fail on the tag mismatch and abort the + // upgrade (the pallet `expect`s success), bricking the chain. Detect it by + // the root's serialized tag — `TypedArenaKey`'s tag embeds the `LedgerState` + // version (`storage-key(midnight:ledger-state[vN]:...)`), so a successful + // tagged decode as a v9 key is a reliable, arena-free discriminator — and + // return the key unchanged with zero synthetic cost. + if tagged_deserialize::, D::Hasher>>(&mut &state_key_v8[..]).is_ok() { + log::info!( + target: LOG_TARGET, + "StateKey already references a ledger-9 state; skipping v8->v9 translation (no-op)" + ); + return Ok((state_key_v8.to_vec(), 0)); + } + + // 1. Decode the v8 arena root and load the v8 ledger wrapper from the arena. + let t_load = std::time::Instant::now(); + let key8: TypedArenaKey, D::Hasher> = tagged_deserialize(&mut &state_key_v8[..]) + .map_err(|e| { + log::error!(target: LOG_TARGET, "failed to deserialize v8 state key: {e:?}"); + LedgerApiError::Deserialization(DeserializationError::TypedArenaKey) + })?; + let ledger8: Sp, D> = default_storage::().arena.get_lazy(&key8).map_err(|e| { + log::error!(target: LOG_TARGET, "failed to load v8 ledger from arena: {e:?}"); + LedgerApiError::NoLedgerState + })?; + log::debug!(target: LOG_TARGET, "[perf] migrate_state_v8_to_v9 load took {:?}", t_load.elapsed()); + + // 2. Run the state translation table over the inner v8 `LedgerState`. + let t_translate = std::time::Instant::now(); + let input: Sp, D> = Sp::new(ledger8.state.clone()); + let mut tl = + TypedTranslationState::, LedgerState9, StateTranslationTable, D>::start( + input, + ) + .map_err(|e| { + log::error!(target: LOG_TARGET, "failed to start v8->v9 translation: {e:?}"); + LedgerApiError::HostApiError + })?; + + let run_budget = CostDuration::from_picoseconds(RUN_BUDGET_PS); + let mut steps = 0usize; + let state9: Sp, D> = loop { + steps += 1; + if steps > MAX_STEPS { + log::error!(target: LOG_TARGET, "v8->v9 translation did not converge in {MAX_STEPS} steps"); + return Err(LedgerApiError::HostApiError); + } + tl = tl.run(run_budget).map_err(|e| { + log::error!(target: LOG_TARGET, "v8->v9 translation step failed: {e:?}"); + LedgerApiError::HostApiError + })?; + if let Some(result) = tl.result().map_err(|e| { + log::error!(target: LOG_TARGET, "v8->v9 translation result failed: {e:?}"); + LedgerApiError::HostApiError + })? { + break result; + } + }; + // Every completed `run` call before the last one must have exhausted its + // full `RUN_BUDGET_PS` — otherwise `tl.result()` would already have been + // `Some` and the loop would have stopped there. Only the final call may + // have used less. `steps * RUN_BUDGET_PS` is therefore a deterministic + // upper bound on the synthetic cost actually consumed, accurate to one + // `RUN_BUDGET_PS` quantum. + let consumed_cost_ps = steps as u64 * RUN_BUDGET_PS; + log::info!( + target: LOG_TARGET, + "v8->v9 ledger state translation complete in {steps} step(s), {:?}, {consumed_cost_ps}ps synthetic cost", + t_translate.elapsed() + ); + + // 3. Wrap the translated state in the v9 ledger wrapper, persist it, and + // flush the arena so the new root is durable before the pallet stores it. + let t_persist = std::time::Instant::now(); + let ledger9 = Ledger9::new((*state9).clone()); + let mut sp9: Sp, D> = default_storage::().arena.alloc(ledger9); + sp9.persist(); + default_storage::().with_backend(|backend| backend.flush_all_changes_to_db()); + log::debug!( + target: LOG_TARGET, + "[perf] migrate_state_v8_to_v9 persist+flush took {:?}", + t_persist.elapsed() + ); + + // 4. Serialize the new v9 arena root for the pallet to store in `StateKey`. + let mut bytes = Vec::new(); + tagged_serialize(&sp9.as_typed_key(), &mut bytes).map_err(|e| { + log::error!(target: LOG_TARGET, "failed to serialize v9 state key: {e:?}"); + LedgerApiError::Serialization(SerializationError::TypedArenaKey) + })?; + + log::debug!(target: LOG_TARGET, "[perf] migrate_state_v8_to_v9 took {:?}", t_total.elapsed()); + Ok((bytes, consumed_cost_ps)) +} + +#[cfg(test)] +mod tests { + use super::*; + use ledger_storage_ledger_8::db::InMemoryDB; + + /// Dev-only: verify that seeding a v8 genesis blob with *this* crate's + /// ledger_8 `Ledger` wrapper reproduces the exact arena root a ledger-8 node + /// (e.g. release 1.0.1) stored in the chain-spec as `genesisStateKey`. If the + /// wrapper serialization drifted, the seeded root wouldn't match and block + /// execution after boot would fail to find the state. Runs only when the two + /// blob paths are provided via env (extracted from a fork-from chain-spec). + #[test] + fn v8_genesis_seed_root_matches_chainspec_key() { + let (Ok(gs_path), Ok(key_path)) = + (std::env::var("HF_GENESIS_STATE"), std::env::var("HF_GENESIS_KEY")) + else { + eprintln!("skipping: set HF_GENESIS_STATE and HF_GENESIS_KEY to run"); + return; + }; + let genesis = std::fs::read(gs_path).expect("read genesis_state"); + let expected = std::fs::read(key_path).expect("read genesisStateKey"); + + let state: LedgerState8 = + midnight_serialize::tagged_deserialize(&mut &genesis[..]) + .expect("deserialize v8 state"); + let ledger = Ledger8::::new(state); + let mut sp = default_storage::().arena.alloc(ledger); + sp.persist(); + let mut got = Vec::new(); + tagged_serialize(&sp.as_typed_key(), &mut got).expect("serialize key"); + + assert_eq!( + got, + expected, + "seeded v8 root must match the chain-spec genesisStateKey \n got={} \n exp={}", + hex::encode(&got), + hex::encode(&expected), + ); + } +} diff --git a/ledger/src/host_api/mod.rs b/ledger/src/host_api/mod.rs index 2b60cdd3a..1b1fd44c3 100644 --- a/ledger/src/host_api/mod.rs +++ b/ledger/src/host_api/mod.rs @@ -14,3 +14,7 @@ pub mod ledger_7; pub mod ledger_8; pub mod ledger_9; + +/// Host-side v8 -> v9 ledger state translation used by the runtime storage migration. +#[cfg(feature = "std")] +pub mod migration_8_to_9; diff --git a/ledger/src/lib.rs b/ledger/src/lib.rs index 1c65c92db..5517115b1 100644 --- a/ledger/src/lib.rs +++ b/ledger/src/lib.rs @@ -145,6 +145,55 @@ pub fn drop_all_default_storage() { ledger_9::storage::drop_default_storage_if_exists(); } +/// Seed the (separate) ledger arena from a genesis `LedgerState` blob, using the +/// deserializer that matches the blob's `ledger-state[vN]` header tag. +/// +/// A node may boot on a chain-spec produced by an older runtime — notably the +/// ledger 8->9 hardfork, where a ledger-9 node starts from a ledger-8 +/// (`ledger-state[v13]`) genesis and only upgrades to v9 later via the runtime +/// migration. Seeding must therefore match the genesis version (the genesis +/// block runs under the old WASM and expects the old-format arena root), not the +/// latest. v8 and v9 share one storage backend, so a v8-seeded arena is exactly +/// what the post-migration v9 runtime reads. Unrecognized tags fall back to the +/// latest version (`ledger_9`), preserving the prior default behaviour. +#[cfg(feature = "std")] +pub fn init_ledger_storage_separate>( + dir: P, + genesis_state: &[u8], + cache_size: usize, +) -> alloc::vec::Vec { + if ledger_8::storage::genesis_matches_this_version(genesis_state) { + ledger_8::storage::init_storage_paritydb_separate(dir, genesis_state, cache_size) + } else { + ledger_9::storage::init_storage_paritydb_separate(dir, genesis_state, cache_size) + } +} + +/// Unified-DB counterpart of [`init_ledger_storage_separate`]. +#[cfg(feature = "std")] +pub fn init_ledger_storage_unified< + D: core::ops::Deref + Default + Send + Sync + 'static, + const COLUMN_OFFSET: u8, +>( + db_instance: D, + genesis_state: &[u8], + cache_size: usize, +) -> alloc::vec::Vec { + if ledger_8::storage::genesis_matches_this_version(genesis_state) { + ledger_8::storage::init_storage_paritydb_unified::( + db_instance, + genesis_state, + cache_size, + ) + } else { + ledger_9::storage::init_storage_paritydb_unified::( + db_instance, + genesis_state, + cache_size, + ) + } +} + mod common; pub mod types { diff --git a/ledger/src/versions/common/mod.rs b/ledger/src/versions/common/mod.rs index 1d2bc61e4..5aca07933 100644 --- a/ledger/src/versions/common/mod.rs +++ b/ledger/src/versions/common/mod.rs @@ -1115,6 +1115,14 @@ where let system_tx = super::system_tx::unlock_to_treasury_system_tx(amount)?; api.tagged_serialize(&system_tx) } + + pub fn construct_distribute_treasury_system_tx( + amount: u128, + ) -> Result, LedgerApiError> { + let api = api::new(); + let system_tx = super::system_tx::distribute_treasury_system_tx(amount)?; + api.tagged_serialize(&system_tx) + } } #[cfg(feature = "std")] diff --git a/ledger/src/versions/common/storage.rs b/ledger/src/versions/common/storage.rs index ef39ea8d9..496757b52 100644 --- a/ledger/src/versions/common/storage.rs +++ b/ledger/src/versions/common/storage.rs @@ -74,6 +74,25 @@ impl core::fmt::Display for GetRootError { } } +/// Returns true if `genesis_state` is a tagged-serialized `LedgerState` of *this* +/// ledger version (i.e. its `midnight:ledger-state[vN]:` header tag matches this +/// version's `LedgerState::tag()`). +/// +/// Used to pick the correct version-specific seeder for the genesis arena when a +/// node boots on a chain-spec produced by an older runtime (e.g. the ledger 8->9 +/// hardfork, where a ledger-9 node starts from a ledger-8 `ledger-state[v13]` +/// genesis before the runtime upgrade migrates it to v9). +#[cfg(feature = "std")] +pub fn genesis_matches_this_version(genesis_state: &[u8]) -> bool { + use super::ledger_storage_local::DefaultDB; + let expected = as Tagged>::tag(); + // `peek_tag` reads the serialized header tag without deserializing the body. + match super::midnight_serialize_local::peek_tag(&mut std::io::Cursor::new(genesis_state)) { + Ok(tag) => tag.as_str() == expected.as_ref(), + Err(_) => false, + } +} + pub fn get_root(state: &[u8], network_id: Option<&str>) -> Result, GetRootError> { // Get empty state key use super::api::Ledger; diff --git a/ledger/src/versions/system_tx/ledger_7.rs b/ledger/src/versions/system_tx/ledger_7.rs index d6c5807c3..292832b0d 100644 --- a/ledger/src/versions/system_tx/ledger_7.rs +++ b/ledger/src/versions/system_tx/ledger_7.rs @@ -37,3 +37,8 @@ pub fn unlock_to_treasury_system_tx(_amount: u128) -> Result bool { false } + +/// Not applicable to ledger-7 (only the ledger-8 bridge exposes this host fn). +pub fn distribute_treasury_system_tx(_amount: u128) -> Result { + Err(LedgerApiError::HostApiError) +} diff --git a/ledger/src/versions/system_tx/ledger_8.rs b/ledger/src/versions/system_tx/ledger_8.rs index d6c5807c3..a893345eb 100644 --- a/ledger/src/versions/system_tx/ledger_8.rs +++ b/ledger/src/versions/system_tx/ledger_8.rs @@ -34,6 +34,14 @@ pub fn unlock_to_treasury_system_tx(_amount: u128) -> Result9 hardfork boundary — the ledger-8 runtime imports the corresponding +/// `construct_distribute_treasury_system_tx` host function (removed for v9). +pub fn distribute_treasury_system_tx(amount: u128) -> Result { + Ok(SystemTransaction::PayBlockRewardsToTreasury { amount }) +} + pub fn is_unlock_to_treasury_system_tx(_tx: &SystemTransaction) -> bool { false } diff --git a/ledger/src/versions/system_tx/ledger_9.rs b/ledger/src/versions/system_tx/ledger_9.rs index 6e854f4a3..99bd7a762 100644 --- a/ledger/src/versions/system_tx/ledger_9.rs +++ b/ledger/src/versions/system_tx/ledger_9.rs @@ -32,3 +32,10 @@ pub fn unlock_to_treasury_system_tx(amount: u128) -> Result bool { matches!(tx, SystemTransaction::UnlockToTreasury { .. }) } + +/// Not applicable to ledger-9: the block-rewards-to-treasury system tx was +/// removed for v9 (only the ledger-8 bridge exposes this host fn, so the ledger-8 +/// WASM can be executed across the 8->9 hardfork). +pub fn distribute_treasury_system_tx(_amount: u128) -> Result { + Err(LedgerApiError::HostApiError) +} diff --git a/node/src/backend/custom_parity_db.rs b/node/src/backend/custom_parity_db.rs index 5cd5ab729..ea4693256 100644 --- a/node/src/backend/custom_parity_db.rs +++ b/node/src/backend/custom_parity_db.rs @@ -109,7 +109,10 @@ pub fn open>( match storage_config.separation { StorageSeparation::Separate => { - midnight_node_ledger::ledger_9::storage::init_storage_paritydb_separate( + // Version-aware: a ledger-9 node may boot on a ledger-8 genesis during + // the 8->9 hardfork; seed the arena with the deserializer matching the + // genesis `ledger-state[vN]` tag (see `init_ledger_storage_separate`). + midnight_node_ledger::init_ledger_storage_separate( &storage_config.db_path, &storage_config.genesis_state, storage_config.cache_size, @@ -117,10 +120,11 @@ pub fn open>( Ok((OwnedDb(db), LedgerStorageDb::SeparateDb(storage_config.db_path.clone()))) }, StorageSeparation::Unified => { - midnight_node_ledger::ledger_9::storage::init_storage_paritydb_unified::< - _, - NUM_COLUMNS_POLKADOT, - >(OwnedDb(db.clone()), &storage_config.genesis_state, storage_config.cache_size); + midnight_node_ledger::init_ledger_storage_unified::<_, NUM_COLUMNS_POLKADOT>( + OwnedDb(db.clone()), + &storage_config.genesis_state, + storage_config.cache_size, + ); Ok((OwnedDb(db.clone()), LedgerStorageDb::UnifiedDb(db.clone()))) }, } diff --git a/pallets/midnight/src/lib.rs b/pallets/midnight/src/lib.rs index a829d4ce0..8a6c36225 100644 --- a/pallets/midnight/src/lib.rs +++ b/pallets/midnight/src/lib.rs @@ -99,7 +99,10 @@ pub mod pallet { } } - const STORAGE_VERSION: StorageVersion = StorageVersion::new(1); + // v2: ledger v8 -> v9 state translation (see `migrations::v2`). A ledger-8 + // runtime is at on-chain version 1; upgrading to this runtime runs the + // `MigrateV1ToV2` translation. Fresh ledger-9 genesis starts at version 2. + const STORAGE_VERSION: StorageVersion = StorageVersion::new(2); // Manually add ~1% of block weight pub const EXTRA_WEIGHT_TX_SIZE: Weight = Weight::from_parts(20_000_000_000, 0); diff --git a/pallets/midnight/src/migrations/mod.rs b/pallets/midnight/src/migrations/mod.rs index bc3f4785d..c7a82f25e 100644 --- a/pallets/midnight/src/migrations/mod.rs +++ b/pallets/midnight/src/migrations/mod.rs @@ -23,3 +23,6 @@ pub const PALLET_MIGRATIONS_ID: &[u8; 19] = b"pallet-midnight-mbm"; // See https://github.com/input-output-hk/midnight-substrate-prototype/pull/382 // for the example of such a migration. // pub mod v1; + +/// Single-block ledger v8 -> v9 state translation (storage version 1 -> 2). +pub mod v2; diff --git a/pallets/midnight/src/migrations/v2.rs b/pallets/midnight/src/migrations/v2.rs new file mode 100644 index 000000000..e9154eca7 --- /dev/null +++ b/pallets/midnight/src/migrations/v2.rs @@ -0,0 +1,117 @@ +// This file is part of midnight-node. +// Copyright (C) 2025-2026 Midnight Foundation +// SPDX-License-Identifier: Apache-2.0 +// Licensed under the Apache License, Version 2.0 (the "License"); +// You may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Storage migration from v1 to v2: ledger-v8 -> ledger-v9 state translation. +//! +//! This is the on-chain half of the ledger 8 -> 9 hardfork. The pallet stores +//! only the ledger state's arena root in [`crate::StateKey`]; the `LedgerState` +//! itself lives in the ledger arena (parity-db). This migration hands the v8 +//! root to the [`migrate_state_v8_to_v9`](midnight_node_ledger::host_api::migration_8_to_9) +//! host function, which walks the v8 state, translates it into the v9 shape +//! (see [`midnight_node_ledger::state_translation_v8_to_v9`]), re-persists it, +//! and returns the new v9 root — which we write back into `StateKey`. +//! +//! It is a single-block migration: the host call translates the whole state in +//! one shot (a few milliseconds for dev/undeployed-sized state). It is wired +//! into [`crate::Migrations`](../../../runtime/src/lib.rs) via +//! [`VersionedMigration`], so it runs at most once, when a runtime at +//! pallet-midnight storage version 1 upgrades to this runtime (storage version +//! 2). A chain whose genesis is already ledger-9 starts at storage version 2 +//! and never runs it. +//! +//! Storage version 1 does not, however, imply the *ledger* state is still v8: +//! the 2.0.0 runtime already ran ledger-9 yet shipped pallet-midnight at storage +//! version 1 (it had no v1->v2 migration), so a network upgrading 2.0.0 -> this +//! runtime triggers this migration over an already-v9 state. The host function +//! detects that case and no-ops (returns the `StateKey` unchanged), so the only +//! effect on that path is the storage-version bump to 2. + +#[cfg(feature = "try-runtime")] +extern crate alloc; + +use crate::{Pallet, StateKey, pallet::Config}; +use frame_support::{ + migrations::VersionedMigration, pallet_prelude::*, traits::UncheckedOnRuntimeUpgrade, +}; +use midnight_node_ledger::types::active_ledger_bridge as LedgerApi; + +#[cfg(feature = "try-runtime")] +use alloc::vec::Vec; + +/// [`UncheckedOnRuntimeUpgrade`] implementation wrapped by [`MigrateV1ToV2`]. +pub struct InnerMigrateV1ToV2(core::marker::PhantomData); + +impl UncheckedOnRuntimeUpgrade for InnerMigrateV1ToV2 { + fn on_runtime_upgrade() -> Weight { + let state_key = StateKey::::get(); + + // The host function reads the v8 arena root, translates the referenced + // `LedgerState` to v9, re-persists it, and returns the new v9 root + // together with the synthetic cost (picoseconds) it charged the + // translation against the ledger's deterministic cost model. If the + // state is already v9 (e.g. a 2.0.0 -> this-runtime upgrade), it no-ops + // and returns the `state_key` unchanged with zero cost. + // A genuine failure here is unrecoverable: the chain would be left + // pointing at a v8 state a ledger-9 runtime cannot read, so we abort the + // upgrade. + let (new_state_key, consumed_cost_ps) = LedgerApi::migrate_state_v8_to_v9(&state_key) + .expect("FATAL: ledger v8->v9 state migration failed"); + + StateKey::::put(new_state_key); + log::info!( + target: "midnight::migration", + "ledger v8->v9 state migration complete; StateKey re-pointed to v9 root ({consumed_cost_ps}ps synthetic cost)" + ); + + // The translation runs natively in the host call, so the pallet-side + // read/write above is negligible next to it; report the ledger's own + // synthetic cost as `ref_time` instead. This is the same 1:1 mapping + // `pallet_midnight::get_tx_weight` uses for ordinary transactions + // (ledger picoseconds -> `Weight` ref_time), and that cost model + // already prices the arena reads/writes the translation performs, so + // no separate `DbWeight` charge is added on top. `proof_size` is 0: + // this chain doesn't build a PoV. Capped at the block's max weight, + // since a pathological state could in principle exceed it. + Weight::from_parts(consumed_cost_ps, 0).min(T::BlockWeights::get().max_block) + } + + #[cfg(feature = "try-runtime")] + fn pre_upgrade() -> Result, sp_runtime::TryRuntimeError> { + frame_support::ensure!( + !StateKey::::get().is_empty(), + "ledger StateKey must be populated before v8->v9 migration" + ); + Ok(Vec::new()) + } + + #[cfg(feature = "try-runtime")] + fn post_upgrade(_state: Vec) -> Result<(), sp_runtime::TryRuntimeError> { + frame_support::ensure!( + !StateKey::::get().is_empty(), + "ledger StateKey must remain populated after v8->v9 migration" + ); + Ok(()) + } +} + +/// Translates the ledger state from v8 to v9 and bumps pallet-midnight storage +/// version 1 -> 2. Wired into the runtime `Migrations` tuple. +pub type MigrateV1ToV2 = VersionedMigration< + 1, + 2, + InnerMigrateV1ToV2, + Pallet, + ::DbWeight, +>; diff --git a/runtime/src/lib.rs b/runtime/src/lib.rs index 5542e4c17..64bebef5b 100644 --- a/runtime/src/lib.rs +++ b/runtime/src/lib.rs @@ -1119,7 +1119,13 @@ pub type Executive = frame_executive::Executive< /// Extrinsic type that has already been checked. pub type CheckedExtrinsic = generic::CheckedExtrinsic; /// Migrations to apply on runtime upgrade. -pub type Migrations = (pallet_throttle::migrations::v1::MigrateV0ToV1,); +pub type Migrations = ( + pallet_throttle::migrations::v1::MigrateV0ToV1, + // Ledger v8 -> v9 state translation (the ledger 8->9 hardfork). Runs once, + // when a ledger-8 runtime (pallet-midnight storage version 1) upgrades to + // this ledger-9 runtime (storage version 2). + pallet_midnight::migrations::v2::MigrateV1ToV2, +); impl frame_system::offchain::CreateTransaction for Runtime where diff --git a/util/toolkit/src/commands/runtime_upgrade.rs b/util/toolkit/src/commands/runtime_upgrade.rs index 2d145167f..367b6daf2 100644 --- a/util/toolkit/src/commands/runtime_upgrade.rs +++ b/util/toolkit/src/commands/runtime_upgrade.rs @@ -14,9 +14,13 @@ // limitations under the License. use std::str::FromStr; +use std::time::Duration; use clap::Args; -use subxt::{OnlineClient, SubstrateConfig, dynamic}; +use subxt::{ + OnlineClient, SubstrateConfig, dynamic, + rpcs::{RpcClient, rpc_params}, +}; use thiserror::Error; use crate::commands::root_call::{self, RootCallArgs}; @@ -35,6 +39,10 @@ pub enum RuntimeUpgradeError { ExtrinsicError(#[from] subxt::error::ExtrinsicError), #[error("transaction finalized error: {0}")] TransactionFinalizedError(#[from] subxt::error::TransactionFinalizedSuccessError), + #[error("transaction progress error: {0}")] + TransactionProgressError(#[from] subxt::error::TransactionProgressError), + #[error("rpc error: {0}")] + RpcError(#[from] subxt::rpcs::Error), #[error("events error: {0}")] EventsError(#[from] subxt::error::EventsError), #[error("keypair parse error: {0}")] @@ -43,6 +51,41 @@ pub enum RuntimeUpgradeError { RootCallError(Box), #[error("runtime upgrade failed: CodeUpdated event not found")] CodeUpdateNotFound, + #[error("timed out waiting for the apply_authorized_upgrade transaction to finalize")] + ApplyFinalizeTimeout, + #[error("runtime upgrade did not enact: spec_version stayed at {0} after applying the code")] + UpgradeNotEnacted(u32), +} + +/// Query the on-chain runtime spec_version via the raw `state_getRuntimeVersion` RPC. +/// +/// We avoid subxt's typed metadata here on purpose: across a runtime upgrade the +/// client's metadata switches to the new runtime, so decoding anything encoded by +/// the old runtime (e.g. the `System.CodeUpdated` event in the apply block) is +/// unreliable. The raw JSON spec_version has no such dependency. +/// (finalized_height, spec_version at the finalized head). +/// +/// We track both because `state_getRuntimeVersion(finalized)` reports the code +/// *stored at* that block — which flips to the new runtime already at the +/// `apply_authorized_upgrade` block, even though that block *executed* under the +/// old runtime (its `MNSV` digest — how the toolkit fetcher classifies a block's +/// ledger version — is still the old spec). The first block that actually +/// executes the new runtime is `apply + 1`. So "spec flipped at the finalized +/// head" is one block early; callers must additionally wait for the finalized +/// height to advance past that point before the fetcher will see a ledger-9 block. +async fn finalized_state(rpc: &RpcClient) -> Result<(u64, u32), RuntimeUpgradeError> { + let hash: serde_json::Value = rpc.request("chain_getFinalizedHead", rpc_params![]).await?; + let header: serde_json::Value = + rpc.request("chain_getHeader", rpc_params![hash.clone()]).await?; + let version: serde_json::Value = + rpc.request("state_getRuntimeVersion", rpc_params![hash]).await?; + let height = header + .get("number") + .and_then(|n| n.as_str()) + .and_then(|s| u64::from_str_radix(s.trim_start_matches("0x"), 16).ok()) + .unwrap_or(0); + let spec = version.get("specVersion").and_then(|v| v.as_u64()).unwrap_or(0) as u32; + Ok((height, spec)) } #[derive(Args)] @@ -78,7 +121,8 @@ pub async fn execute(args: RuntimeUpgradeArgs) -> Result<(), RuntimeUpgradeError log::info!("Code hash: 0x{}", hex::encode(code_hash)); // Step 3: Build System::authorize_upgrade call and encode it - let api = OnlineClient::::from_insecure_url(&args.rpc_url).await?; + let rpc_client = RpcClient::from_insecure_url(&args.rpc_url).await?; + let api = OnlineClient::::from_rpc_client(rpc_client.clone()).await?; let authorize_upgrade_call = dynamic::tx("System", "authorize_upgrade", vec![dynamic::Value::from_bytes(&code_hash)]); let encoded_call = api.tx().await?.call_data(&authorize_upgrade_call)?; @@ -101,28 +145,54 @@ pub async fn execute(args: RuntimeUpgradeArgs) -> Result<(), RuntimeUpgradeError let apply_upgrade_call = dynamic::tx("System", "apply_authorized_upgrade", vec![dynamic::Value::from_bytes(&code)]); - let apply_events = api - .tx() - .await? - .sign_and_submit_then_watch_default(&apply_upgrade_call, &signer) - .await? - .wait_for_finalized_success() - .await?; - - // Step 6: Verify CodeUpdated event - let mut success = false; - for event in apply_events.iter() { - let event = event?; - if event.pallet_name() == "System" && event.event_name() == "CodeUpdated" { - log::info!("Code update success: {:?}", event); - success = true; - break; + let (_, pre_spec_version) = finalized_state(&rpc_client).await?; + log::info!("Pre-upgrade spec_version (finalized): {pre_spec_version}"); + + // Wait only for finalization — NOT `wait_for_finalized_success`, which eagerly + // decodes the block's events. `apply_authorized_upgrade` swaps the on-chain + // code, and subxt's metadata follows it to the new runtime, so decoding the + // old runtime's `System.CodeUpdated` event in that block fails. Confirm the + // upgrade succeeded by observing the spec_version bump below instead. + let submit = async { + api.tx() + .await? + .sign_and_submit_then_watch_default(&apply_upgrade_call, &signer) + .await? + .wait_for_finalized() + .await + .map_err(RuntimeUpgradeError::from) + }; + tokio::time::timeout(Duration::from_secs(120), submit) + .await + .map_err(|_| RuntimeUpgradeError::ApplyFinalizeTimeout)??; + + // Step 6: Confirm the upgrade is not just applied but *executing* at a + // finalized block. `state_getRuntimeVersion(finalized)` reports the stored + // code, which flips to the new runtime already at the apply block — but that + // block's `MNSV` execution-version digest (how the fetcher classifies it) is + // still the old spec. The first block that runs the new runtime is apply+1. + // So: note the finalized height where the stored spec first exceeds pre, then + // wait for the finalized height to advance past it (apply+1 finalized). Only + // then will a downstream `fetch` see a ledger-9-classified block. + let mut flip_height: Option = None; + for _ in 0..60 { + tokio::time::sleep(Duration::from_secs(3)).await; + let (height, spec) = finalized_state(&rpc_client).await?; + if spec > pre_spec_version { + match flip_height { + None => flip_height = Some(height), + Some(h0) if height > h0 => { + log::info!( + "Runtime upgrade completed successfully! spec_version {pre_spec_version} -> {spec}; \ + new runtime executing since finalized #{}, now finalized #{height}", + h0 + 1, + ); + return Ok(()); + }, + _ => {}, + } } } - if !success { - return Err(RuntimeUpgradeError::CodeUpdateNotFound); - } - log::info!("Runtime upgrade completed successfully!"); - Ok(()) + Err(RuntimeUpgradeError::UpgradeNotEnacted(pre_spec_version)) } diff --git a/util/toolkit/src/tx_generator/builder/mod.rs b/util/toolkit/src/tx_generator/builder/mod.rs index 9ce2f9558..b1ec9adb3 100644 --- a/util/toolkit/src/tx_generator/builder/mod.rs +++ b/util/toolkit/src/tx_generator/builder/mod.rs @@ -19,7 +19,7 @@ use midnight_node_ledger_helpers::fork::{ fork_aware_context::{ ForkAwareLedgerContext, apply_block_7, apply_block_8, apply_block_9, block_context_from_raw_7, block_context_from_raw_8, block_context_from_raw_9, - fork_context_7_to_8, + fork_context_7_to_8, fork_context_8_to_9, }, raw_block_data::{LedgerVersion, RawBlockData}, }; @@ -1110,6 +1110,25 @@ fn replay_blocks_9( } } +/// Fork a ledger-8 context to ledger 9 (real state translation) and replay the +/// ledger-9 blocks, if any. Returns the ledger-8 context unchanged when there are +/// no ledger-9 blocks. +fn fork_8_to_9_if_needed( + ctx8: midnight_node_ledger_helpers::ledger_8::context::LedgerContext, + l9_blocks: &[RawBlockData], + cached: &[(WalletSeed, CachedWalletState)], + schemes: &WalletSchemes, +) -> ForkAwareLedgerContext { + if l9_blocks.is_empty() { + ForkAwareLedgerContext::Ledger8(ctx8) + } else { + let ctx9 = + timed!("fork_context_8_to_9", fork_context_8_to_9(ctx8)).expect("fork 8 to 9 failed"); + replay_blocks_9(&ctx9, l9_blocks, cached, schemes); + ForkAwareLedgerContext::Ledger9(ctx9) + } +} + /// Replays blocks across a potential Ledger7→Ledger8->Ledger9 fork boundaries, /// injecting cached wallets at their saved height. pub(crate) fn replay_blocks( @@ -1133,27 +1152,27 @@ pub(crate) fn replay_blocks( l8_and_l9_blocks.partition_point(|b| b.ledger_version() == LedgerVersion::Ledger8); let (l8_blocks, l9_blocks) = l8_and_l9_blocks.split_at(fork_8_to_9_idx); - assert!( - l9_blocks.is_empty() || (l7_blocks.is_empty() && l8_blocks.is_empty()), - "chain has Ledger9 blocks and eariler version blocks. This is not supported yet!" - ); - + // Replay each version's blocks in order, forking the context across the + // 7->8 and 8->9 boundaries as needed. The 8->9 fork performs a real state + // translation (see `fork_context_8_to_9`) so post-hardfork transactions are + // built at ledger 9, matching the upgraded chain. let result = match fork_ctx { ForkAwareLedgerContext::Ledger7(ctx7) => { replay_blocks_7(&ctx7, l7_blocks); - if l8_blocks.is_empty() { - assert!(cached.is_empty(), "cached wallets with no Ledger8 blocks"); + if l8_blocks.is_empty() && l9_blocks.is_empty() { + assert!(cached.is_empty(), "cached wallets with no Ledger8/9 blocks"); ForkAwareLedgerContext::Ledger7(ctx7) } else { - let ctx8 = fork_context_7_to_8(ctx7).expect("fork 7 to 8 failed"); + let ctx8 = timed!("fork_context_7_to_8", fork_context_7_to_8(ctx7)) + .expect("fork 7 to 8 failed"); replay_blocks_8(&ctx8, l8_blocks); - ForkAwareLedgerContext::Ledger8(ctx8) + fork_8_to_9_if_needed(ctx8, l9_blocks, cached, schemes) } }, ForkAwareLedgerContext::Ledger8(ctx8) => { assert!(l7_blocks.is_empty(), "Ledger7 blocks with Ledger8 context"); replay_blocks_8(&ctx8, l8_blocks); - ForkAwareLedgerContext::Ledger8(ctx8) + fork_8_to_9_if_needed(ctx8, l9_blocks, cached, schemes) }, ForkAwareLedgerContext::Ledger9(ctx9) => { assert!(l7_blocks.is_empty(), "Ledger7 blocks with Ledger9 context"); diff --git a/util/toolkit/test-images.docker-compose.yml b/util/toolkit/test-images.docker-compose.yml index 42cdd834f..71a079559 100644 --- a/util/toolkit/test-images.docker-compose.yml +++ b/util/toolkit/test-images.docker-compose.yml @@ -15,8 +15,11 @@ services: depends_on: guard: condition: service_completed_successfully + # Ledger-8 release: the hardfork_e2e test forks from this (ledger 8, runtime + # spec_version 1_000_000, pallet-midnight storage version 1) up to the current + # ledger-9 runtime, exercising the v8->v9 state migration. midnight-node-fork-from: - image: ${FORK_FROM_NODE_IMAGE:-midnightntwrk/midnight-node:0.21.0} + image: ${FORK_FROM_NODE_IMAGE:-midnightntwrk/midnight-node:1.0.1} depends_on: guard: condition: service_completed_successfully diff --git a/util/toolkit/tests/hardfork_e2e.rs b/util/toolkit/tests/hardfork_e2e.rs index 578788807..8507a90ac 100644 --- a/util/toolkit/tests/hardfork_e2e.rs +++ b/util/toolkit/tests/hardfork_e2e.rs @@ -54,7 +54,6 @@ async fn run_cli(args: &[&str]) { } #[test_log::test(tokio::test)] -#[ignore = "Migration to Ledger v9 is not yet supported. Issue #1580."] async fn hardfork_single_tx() { // 1. Generate chain-spec from fork-from node let (old_name, old_tag) = test_image("midnight-node-fork-from"); From 41196f7e3019b373479ba3ee3709f3c1229df3e2 Mon Sep 17 00:00:00 2001 From: Oscar Bailey <79094698+ozgb@users.noreply.github.com> Date: Wed, 12 Aug 2026 12:11:08 +0100 Subject: [PATCH 02/13] fix(toolkit): drop WalletSchemes from the 8->9 fork replay path #1925 threads a `&WalletSchemes` through `fork_8_to_9_if_needed` and `replay_blocks_9`. That parameter comes from the ECDSA work on main (#1837, #1861), which is not on release/node-2.0.0, so the cherry-pick merges textually clean and only fails at compile. Drop the parameter and its three call sites; wallet-scheme selection does not exist on this branch and nothing else in the fork path reads it. Adaptation of the cherry-pick, not an upstream change. Assisted-by: Claude:claude-opus-5 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> --- util/toolkit/src/tx_generator/builder/mod.rs | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/util/toolkit/src/tx_generator/builder/mod.rs b/util/toolkit/src/tx_generator/builder/mod.rs index b1ec9adb3..60161154e 100644 --- a/util/toolkit/src/tx_generator/builder/mod.rs +++ b/util/toolkit/src/tx_generator/builder/mod.rs @@ -1117,14 +1117,13 @@ fn fork_8_to_9_if_needed( ctx8: midnight_node_ledger_helpers::ledger_8::context::LedgerContext, l9_blocks: &[RawBlockData], cached: &[(WalletSeed, CachedWalletState)], - schemes: &WalletSchemes, ) -> ForkAwareLedgerContext { if l9_blocks.is_empty() { ForkAwareLedgerContext::Ledger8(ctx8) } else { let ctx9 = timed!("fork_context_8_to_9", fork_context_8_to_9(ctx8)).expect("fork 8 to 9 failed"); - replay_blocks_9(&ctx9, l9_blocks, cached, schemes); + replay_blocks_9(&ctx9, l9_blocks, cached); ForkAwareLedgerContext::Ledger9(ctx9) } } @@ -1166,13 +1165,13 @@ pub(crate) fn replay_blocks( let ctx8 = timed!("fork_context_7_to_8", fork_context_7_to_8(ctx7)) .expect("fork 7 to 8 failed"); replay_blocks_8(&ctx8, l8_blocks); - fork_8_to_9_if_needed(ctx8, l9_blocks, cached, schemes) + fork_8_to_9_if_needed(ctx8, l9_blocks, cached) } }, ForkAwareLedgerContext::Ledger8(ctx8) => { assert!(l7_blocks.is_empty(), "Ledger7 blocks with Ledger8 context"); replay_blocks_8(&ctx8, l8_blocks); - fork_8_to_9_if_needed(ctx8, l9_blocks, cached, schemes) + fork_8_to_9_if_needed(ctx8, l9_blocks, cached) }, ForkAwareLedgerContext::Ledger9(ctx9) => { assert!(l7_blocks.is_empty(), "Ledger7 blocks with Ledger9 context"); From e940464ffe2b6b38b3486eb20eeb7aa330b1fc29 Mon Sep 17 00:00:00 2001 From: Oscar Bailey <79094698+ozgb@users.noreply.github.com> Date: Wed, 12 Aug 2026 12:11:24 +0100 Subject: [PATCH 03/13] fix: backport ledger version bump (#2001) Signed-off-by: Giles Cope Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> # Conflicts: # Cargo.lock # Cargo.toml Co-authored-by: Giles Cope (cherry picked from commit 6fb0cc04e23f8129b60d01a3cef0f93b5b325526) Assisted-by: Claude:claude-opus-5 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> --- Cargo.lock | 74 +++++++++++++++++++++++++++--------------------------- Cargo.toml | 8 +++--- 2 files changed, 41 insertions(+), 41 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index dea0ae423..415d954d8 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2029,7 +2029,7 @@ checksum = "af491d569909a7e4dee0ad7db7f5341fef5c614d5b8ec8cf765732aba3cff681" dependencies = [ "serde", "termcolor", - "unicode-width 0.1.14", + "unicode-width 0.2.2", ] [[package]] @@ -2044,7 +2044,7 @@ version = "3.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "faf9468729b8cbcea668e36183cb69d317348c2e08e994829fb56ebfdfbaac34" dependencies = [ - "windows-sys 0.48.0", + "windows-sys 0.61.2", ] [[package]] @@ -3064,7 +3064,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7ab67060fc6b8ef687992d439ca0fa36e7ed17e9a0b16b25b601e8757df720de" dependencies = [ "data-encoding", - "syn 1.0.109", + "syn 2.0.117", ] [[package]] @@ -5648,7 +5648,7 @@ dependencies = [ "libc", "percent-encoding", "pin-project-lite", - "socket2 0.5.10", + "socket2 0.6.3", "system-configuration 0.7.0", "tokio", "tower-service", @@ -7791,9 +7791,9 @@ dependencies = [ [[package]] name = "midnight-ledger" -version = "8.1.0" +version = "8.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2182054f3a43ccabac514448fff2487437be293f517a01afc23905df701e8548" +checksum = "479798394847a6686bb9fabda270d16f8525457e9adcc9d2c3486b2422cc4e1c" dependencies = [ "anyhow", "derive-where", @@ -7809,9 +7809,9 @@ dependencies = [ "midnight-ledger-static", "midnight-onchain-runtime 3.1.0", "midnight-serialize", - "midnight-storage 2.0.1", + "midnight-storage 2.0.2", "midnight-transient-crypto 2.2.0", - "midnight-zswap 8.1.0", + "midnight-zswap 8.1.1", "rand 0.8.5", "rayon", "serde", @@ -7852,7 +7852,7 @@ dependencies = [ "midnight-ledger-static", "midnight-onchain-runtime 4.0.0", "midnight-serialize", - "midnight-storage 2.0.1", + "midnight-storage 2.0.2", "midnight-transient-crypto 2.2.0", "midnight-transient-crypto 3.0.0", "midnight-zkir", @@ -8042,7 +8042,7 @@ dependencies = [ "midnight-coin-structure 2.0.1", "midnight-coin-structure 3.0.0", "midnight-ledger 7.0.3", - "midnight-ledger 8.1.0", + "midnight-ledger 8.1.1", "midnight-ledger-v9", "midnight-node-ledger-helpers", "midnight-node-res", @@ -8054,12 +8054,12 @@ dependencies = [ "midnight-primitives-ledger", "midnight-serialize", "midnight-storage 1.1.1", - "midnight-storage 2.0.1", + "midnight-storage 2.0.2", "midnight-storage-core", "midnight-transient-crypto 2.2.0", "midnight-transient-crypto 3.0.0", "midnight-zswap 7.0.3", - "midnight-zswap 8.1.0", + "midnight-zswap 8.1.1", "midnight-zswap 9.0.0", "moka 0.11.3", "parity-db 0.5.4", @@ -8095,7 +8095,7 @@ dependencies = [ "midnight-coin-structure 2.0.1", "midnight-coin-structure 3.0.0", "midnight-ledger 7.0.3", - "midnight-ledger 8.1.0", + "midnight-ledger 8.1.1", "midnight-ledger-v9", "midnight-onchain-runtime 2.0.1", "midnight-onchain-runtime 3.1.0", @@ -8104,12 +8104,12 @@ dependencies = [ "midnight-onchain-state 4.0.0", "midnight-serialize", "midnight-storage 1.1.1", - "midnight-storage 2.0.1", + "midnight-storage 2.0.2", "midnight-transient-crypto 2.2.0", "midnight-transient-crypto 3.0.0", "midnight-zkir", "midnight-zswap 7.0.3", - "midnight-zswap 8.1.0", + "midnight-zswap 8.1.1", "midnight-zswap 9.0.0", "rand 0.8.5", "rayon", @@ -8330,7 +8330,7 @@ dependencies = [ "midnight-onchain-state 3.0.0", "midnight-onchain-vm 3.1.0", "midnight-serialize", - "midnight-storage 2.0.1", + "midnight-storage 2.0.2", "midnight-transient-crypto 2.2.0", "rand 0.8.5", "serde", @@ -8355,7 +8355,7 @@ dependencies = [ "midnight-onchain-state 4.0.0", "midnight-onchain-vm 4.0.0", "midnight-serialize", - "midnight-storage 2.0.1", + "midnight-storage 2.0.2", "midnight-transient-crypto 3.0.0", "rand 0.8.5", "serde", @@ -8394,7 +8394,7 @@ dependencies = [ "midnight-base-crypto", "midnight-coin-structure 2.0.1", "midnight-serialize", - "midnight-storage 2.0.1", + "midnight-storage 2.0.2", "midnight-transient-crypto 2.2.0", "rand 0.8.5", "serde", @@ -8412,7 +8412,7 @@ dependencies = [ "midnight-base-crypto", "midnight-coin-structure 3.0.0", "midnight-serialize", - "midnight-storage 2.0.1", + "midnight-storage 2.0.2", "midnight-transient-crypto 2.2.0", "midnight-transient-crypto 3.0.0", "rand 0.8.5", @@ -8456,7 +8456,7 @@ dependencies = [ "midnight-coin-structure 2.0.1", "midnight-onchain-state 3.0.0", "midnight-serialize", - "midnight-storage 2.0.1", + "midnight-storage 2.0.2", "midnight-transient-crypto 2.2.0", "rand 0.8.5", "rpds 1.2.1", @@ -8477,7 +8477,7 @@ dependencies = [ "midnight-coin-structure 3.0.0", "midnight-onchain-state 4.0.0", "midnight-serialize", - "midnight-storage 2.0.1", + "midnight-storage 2.0.2", "midnight-transient-crypto 3.0.0", "rand 0.8.5", "rpds 1.2.1", @@ -8703,9 +8703,9 @@ dependencies = [ [[package]] name = "midnight-storage" -version = "2.0.1" +version = "2.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "210e601a89aee79ce2b007cf96c1a56c23baa306b0c40b9b98d12c27e18d1981" +checksum = "13f3f6a6f45732db644df3eb723a9fc3877490d47f937ea833609ed6cb1508cf" dependencies = [ "crypto", "derive-where", @@ -8935,9 +8935,9 @@ dependencies = [ [[package]] name = "midnight-zswap" -version = "8.1.0" +version = "8.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4c83f946d8ac03abea38ca470599801fa08e91e2ddf3cb0963027a7701180c7c" +checksum = "07ff4a06f9d1dcb857be73dd926fcb233b61af3d2a8ab0103b5e39725cd4feb8" dependencies = [ "derive-where", "fake", @@ -8950,7 +8950,7 @@ dependencies = [ "midnight-ledger-static", "midnight-onchain-runtime 3.1.0", "midnight-serialize", - "midnight-storage 2.0.1", + "midnight-storage 2.0.2", "midnight-transient-crypto 2.2.0", "rand 0.8.5", "serde", @@ -8975,7 +8975,7 @@ dependencies = [ "midnight-ledger-static", "midnight-onchain-runtime 4.0.0", "midnight-serialize", - "midnight-storage 2.0.1", + "midnight-storage 2.0.2", "midnight-transient-crypto 2.2.0", "midnight-transient-crypto 3.0.0", "midnight-zkir", @@ -9883,7 +9883,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7d8fae84b431384b68627d0f9b3b1245fcf9f46f6c0e3dc902e9dce64edd1967" dependencies = [ "libc", - "windows-sys 0.45.0", + "windows-sys 0.61.2", ] [[package]] @@ -12432,8 +12432,8 @@ version = "0.13.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "be769465445e8c1474e9c5dac2018218498557af32d9ed057325ec9a41ae81bf" dependencies = [ - "heck 0.4.1", - "itertools 0.10.5", + "heck 0.5.0", + "itertools 0.14.0", "log", "multimap", "once_cell", @@ -12452,8 +12452,8 @@ version = "0.14.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "343d3bd7056eda839b03204e68deff7d1b13aba7af2b2fd16890697274262ee7" dependencies = [ - "heck 0.4.1", - "itertools 0.10.5", + "heck 0.5.0", + "itertools 0.14.0", "log", "multimap", "petgraph 0.8.3", @@ -12485,7 +12485,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8a56d757972c98b346a9b766e3f02746cde6dd1cd1d1d563472929fdd74bec4d" dependencies = [ "anyhow", - "itertools 0.10.5", + "itertools 0.14.0", "proc-macro2", "quote 1.0.45", "syn 2.0.117", @@ -12498,7 +12498,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "27c6023962132f4b30eb4c172c91ce92d933da334c59c23cddee82358ddafb0b" dependencies = [ "anyhow", - "itertools 0.10.5", + "itertools 0.14.0", "proc-macro2", "quote 1.0.45", "syn 2.0.117", @@ -12681,7 +12681,7 @@ dependencies = [ "quinn-udp", "rustc-hash 2.1.2", "rustls", - "socket2 0.5.10", + "socket2 0.6.3", "thiserror 2.0.18", "tokio", "tracing", @@ -12719,7 +12719,7 @@ dependencies = [ "cfg_aliases 0.2.1", "libc", "once_cell", - "socket2 0.5.10", + "socket2 0.6.3", "tracing", "windows-sys 0.60.2", ] @@ -20645,7 +20645,7 @@ version = "0.1.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" dependencies = [ - "windows-sys 0.48.0", + "windows-sys 0.61.2", ] [[package]] diff --git a/Cargo.toml b/Cargo.toml index 700f473bd..91634ce44 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -85,17 +85,17 @@ zswap = { version = "=7.0.3", package = "midnight-zswap" } zkir = { version = "^2.2.0", package = "midnight-zkir" } # Ledger 8 (compatible with layout-v2) -# coin-structure and transient-crypto (2.x) share versions with L7 so reuse those entries. -mn-ledger-8 = { version = "=8.1.0", package = "midnight-ledger" } +# coin-structure, transient-crypto and zkir share versions with L7 so reuse those entries. +mn-ledger-8 = { version = "=8.1.1", package = "midnight-ledger" } # `state-translation` (needed by the v8->v9 storage migration) pulls in # `public-internal-structure`, exposing `merkle_patricia_trie`/`storable`/the # `state_translation` module used by `midnight_node_ledger::state_translation_v8_to_v9`. -ledger-storage-ledger-8 = { version = "=2.0.1", package = "midnight-storage", features = ["parity-db", "state-translation"] } +ledger-storage-ledger-8 = { version = "=2.0.2", package = "midnight-storage", features = ["parity-db", "state-translation"] } onchain-runtime-ledger-8 = { version = "=3.1.0", package = "midnight-onchain-runtime" } # Same `midnight-onchain-state` instance that `mn-ledger-8`'s `ContractState` # resolves to (via onchain-runtime 3.1.0); used as the v8 side of the state translation table. onchain-state-ledger-8 = { version = "=3.0.0", package = "midnight-onchain-state" } -zswap-ledger-8 = { version = "=8.1.0", package = "midnight-zswap" } +zswap-ledger-8 = { version = "=8.1.1", package = "midnight-zswap" } # Ledger 9 (compatible with layout-v2; midnight-ledger-v9 crate) # storage shares versions with L8 so reuses that entry. coin-structure and From 75019d45086f2d31628e8151a6ae1112eb476b69 Mon Sep 17 00:00:00 2001 From: Oscar Bailey <79094698+ozgb@users.noreply.github.com> Date: Wed, 12 Aug 2026 12:11:25 +0100 Subject: [PATCH 04/13] fix(ledger): serve ledger state reads at the ledger-hardfork set_code block (#1985) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(ledger): serve ledger state reads at the ledger-hardfork set_code block On a chain that hardforks ledger 8 -> 9 via a governance `set_code`, exactly one historical block is permanently unreadable: every ledger state read at that hash fails with `Deserialization(TypedArenaKey)` (GH #1959). The pre-fork runtime ships `system_version: 1`, so `frame_system` overwrites `:code` *inside* the `set_code` block, while pallet-midnight's v8 -> v9 state translation only runs in the next block's `initialize_block`. That block's committed state therefore pairs ledger-9 `:code` with a ledger-8 `StateKey` forever, and reading it executes ledger-9 code against a ledger-8 arena root. The read-only accessors of the ledger-9 host API now check the tagged-serialization header of the `state_key` they are handed and, when it is a ledger-8 arena root, serve the read from the ledger-8 bridge — which is what the state at that block actually is. v8 and v9 share one storage crate and hence one arena, so this is a pure dispatch with no data movement. The `StateKey` tag is the signal rather than the pallet storage version, because the 2.0.0 runtime already ran ledger 9 while still reporting pallet-midnight storage version 1. Guarded accessors: get_contract_state, get_zswap_chain_state, get_zswap_state_root, get_ledger_state_root, get_ledger_parameters, get_c_to_m_bridge_min_amount, get_unclaimed_amount, get_bridge_receiving_amount. Placing the dispatch in the host function rather than in the node's JSON-RPC layer covers every caller at once: the `midnight_*` RPCs, `MidnightRuntimeApi` through `state_call`, and subxt-based tooling such as chain-indexer (GH #1969), none of which touch the node's RPC handlers. The transaction paths are deliberately left alone — at the skew block they concern ledger-9-format transactions that ledger-8 code cannot deserialize in any case, and they resolve on their own one block later; this is documented at `get_transaction_cost`. hardfork_e2e now walks the fork boundary and asserts both the `midnight_*` RPCs and a raw `state_call` answer at `applied - 1`, `applied` and `applied + 1`. Assisted-by: Claude:claude-opus-5 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> * chore: point change file at the new PR Assisted-by: Claude:claude-opus-5 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> * docs: tighten doc comment Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> * docs: newline Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> --------- Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> (cherry picked from commit 54dffa05f959495f605f43ff76540bbce4905731) Assisted-by: Claude:claude-opus-5 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> --- .../hardfork-skew-block-ledger-reads.md | 36 +++++ ledger/src/host_api/ledger_9.rs | 113 +++++++++++++++ ledger/src/lib.rs | 44 ++++++ util/toolkit/tests/hardfork_e2e.rs | 135 +++++++++++++++++- 4 files changed, 327 insertions(+), 1 deletion(-) create mode 100644 changes/node/changed/hardfork-skew-block-ledger-reads.md diff --git a/changes/node/changed/hardfork-skew-block-ledger-reads.md b/changes/node/changed/hardfork-skew-block-ledger-reads.md new file mode 100644 index 000000000..d03b2d27b --- /dev/null +++ b/changes/node/changed/hardfork-skew-block-ledger-reads.md @@ -0,0 +1,36 @@ +#node #ledger +# Serve ledger state reads at the ledger-hardfork `set_code` block + +On a chain that hardforks ledger 8 -> 9 via a governance `set_code`, exactly one historical +block was permanently unreadable: every ledger state read at that hash failed with +`Deserialization(TypedArenaKey)`. The pre-fork runtime shipped `system_version: 1`, so +`frame_system` overwrote `:code` *inside* the `set_code` block while pallet-midnight's v8 -> v9 +state translation only ran in the next block's `initialize_block`. That block's committed state +therefore pairs ledger-9 `:code` with a ledger-8 `StateKey` forever, and any read at that hash +executes ledger-9 code against a ledger-8 arena root. + +The read-only accessors of the ledger-9 host API now check the tagged-serialization header of +the `state_key` they are handed. If it is a ledger-8 arena root, the read is served from the +ledger-8 bridge instead — v8 and v9 share one storage crate and hence one arena, so this is a +pure dispatch with no data movement. The `StateKey` tag is the signal rather than the pallet +storage version, because the 2.0.0 runtime already ran ledger 9 while still reporting +pallet-midnight storage version 1. + +Because the dispatch sits in the host function, it covers every caller of the affected reads — +the `midnight_*` JSON-RPCs, `MidnightRuntimeApi` through `state_call`, and subxt-based tooling +such as `chain-indexer` — rather than only the node's own RPC layer. Affected reads: +`get_contract_state`, `get_zswap_chain_state`, `get_zswap_state_root`, `get_ledger_state_root`, +`get_ledger_parameters`, `get_c_to_m_bridge_min_amount`, `get_unclaimed_amount`, +`get_bridge_receiving_amount`. + +Not covered, deliberately: the transaction paths (`get_transaction_cost`, +`validate_transaction`, `apply_transaction`). At the skew block those concern ledger-9-format +transactions, which ledger-8 code cannot deserialize in any case, and they resolve on their own +one block later once the migration has run. + +Behaviour note: at the `set_code` block `midnight_ledgerStateRoot` now returns a **v8-tagged** +root — correct, since that block's state *is* v8 — so consumers walking the fork see the tag +flip at the migration block rather than a hole. + +PR: https://github.com/midnightntwrk/midnight-node/pull/1985 +Issue: https://github.com/midnightntwrk/midnight-node/issues/1959 diff --git a/ledger/src/host_api/ledger_9.rs b/ledger/src/host_api/ledger_9.rs index 2e1ca7d61..982082939 100644 --- a/ledger/src/host_api/ledger_9.rs +++ b/ledger/src/host_api/ledger_9.rs @@ -45,6 +45,52 @@ fn is_unified(mut ext: &mut dyn Externalities) -> bool { ) } +#[cfg(feature = "std")] +use crate::ledger_8::Bridge as Bridge8; +#[cfg(feature = "std")] +type Signature8 = crate::ledger_8::TransactionSignature; + +/// Translate a ledger-8 `LedgerApiError` into its ledger-9 counterpart. +/// +/// The two are distinct types generated from the same source +/// (`versions/common/types.rs`) by module parameterization, so their SCALE +/// encodings are identical by construction. Round-tripping keeps this correct +/// when a variant is added, where a hand-written match would need editing in +/// lockstep. `ledger_8_error_encoding_matches_ledger_9` guards the assumption. +#[cfg(feature = "std")] +fn as_ledger_9_error(error: crate::ledger_8::types::LedgerApiError) -> LedgerApiError { + use parity_scale_codec::{Decode, Encode}; + LedgerApiError::decode(&mut &error.encode()[..]).unwrap_or(LedgerApiError::HostApiError) +} + +/// Serve a read-only ledger accessor from the ledger-8 bridge when `$state_key` +/// is a ledger-8 arena root, by returning early from the enclosing host function. +/// Falls through to the ledger-9 body otherwise. +/// +/// For the ledger 8 -> 9 hard-fork, `system_version == 1` which means runtime code +/// is applied during the upgrade block rather than queued to be applied in the next +/// block. This code allows off-chain runtime calls to access historic block data +/// using the correct ledger api despite the runtime code/chain data skew. +/// +/// This will not be needed for future forks; see: +/// - https://github.com/midnightntwrk/midnight-node/pull/1900 +/// +/// `$call` names the `Bridge` method and takes its arguments verbatim; only the +/// storage-mode dispatch and the error translation are supplied here. +#[cfg(feature = "std")] +macro_rules! serve_pre_migration_v8_read { + ($ext:expr, $state_key:expr, $call:ident($($arg:expr),* $(,)?)) => { + if crate::is_ledger_8_state_key($state_key) { + let result = if is_unified($ext) { + Bridge8::::$call($($arg),*) + } else { + Bridge8::::$call($($arg),*) + }; + return result.map_err(as_ledger_9_error); + } + }; +} + #[runtime_interface] pub trait Ledger9Bridge { fn set_default_storage(&mut self) { @@ -219,6 +265,12 @@ pub trait Ledger9Bridge { state_key: PassFatPointerAndRead<&[u8]>, contract_address: PassFatPointerAndRead<&[u8]>, ) -> AllocateAndReturnByCodec, LedgerApiError>> { + serve_pre_migration_v8_read!( + *self, + state_key, + get_contract_state(state_key, contract_address) + ); + if is_unified(*self) { Bridge::::get_contract_state(state_key, contract_address) } else { @@ -250,6 +302,12 @@ pub trait Ledger9Bridge { state_key: PassFatPointerAndRead<&[u8]>, contract_address: PassFatPointerAndRead<&[u8]>, ) -> AllocateAndReturnByCodec, LedgerApiError>> { + serve_pre_migration_v8_read!( + *self, + state_key, + get_zswap_chain_state(state_key, contract_address) + ); + if is_unified(*self) { Bridge::::get_zswap_chain_state(state_key, contract_address) } else { @@ -266,6 +324,12 @@ pub trait Ledger9Bridge { state_key: PassFatPointerAndRead<&[u8]>, beneficiary: PassFatPointerAndRead<&[u8]>, ) -> AllocateAndReturnByCodec> { + serve_pre_migration_v8_read!( + *self, + state_key, + get_unclaimed_amount(state_key, beneficiary) + ); + if is_unified(*self) { Bridge::::get_unclaimed_amount(state_key, beneficiary) } else { @@ -281,6 +345,8 @@ pub trait Ledger9Bridge { &mut self, state_key: PassFatPointerAndRead<&[u8]>, ) -> AllocateAndReturnByCodec, LedgerApiError>> { + serve_pre_migration_v8_read!(*self, state_key, get_ledger_parameters(state_key)); + if is_unified(*self) { Bridge::::get_ledger_parameters(state_key) } else { @@ -296,6 +362,8 @@ pub trait Ledger9Bridge { &mut self, state_key: PassFatPointerAndRead<&[u8]>, ) -> AllocateAndReturnByCodec> { + serve_pre_migration_v8_read!(*self, state_key, get_c_to_m_bridge_min_amount(state_key)); + if is_unified(*self) { Bridge::::get_c_to_m_bridge_min_amount(state_key) } else { @@ -305,6 +373,14 @@ pub trait Ledger9Bridge { /* * Returns the expected fee to pay for a submitting a transaction + * + * No `serve_pre_migration_v8_read!` guard here, unlike the accessors above: a + * cost estimate is always requested for a transaction about to be submitted, + * and `get_ledger_version` reports ledger 9 as soon as the new code is live, so + * `tx` is a v9-format transaction that ledger-8 code cannot deserialize anyway. + * The same reasoning covers the transaction paths (`validate_transaction`, + * `apply_transaction`, ...): at the skew block they concern v9 transactions, and + * they resolve on their own one block later once the migration has run. */ fn get_transaction_cost( &mut self, @@ -338,6 +414,8 @@ pub trait Ledger9Bridge { &mut self, state_key: PassFatPointerAndRead<&[u8]>, ) -> AllocateAndReturnByCodec, LedgerApiError>> { + serve_pre_migration_v8_read!(*self, state_key, get_zswap_state_root(state_key)); + if is_unified(*self) { Bridge::::get_zswap_state_root(state_key) } else { @@ -360,6 +438,8 @@ pub trait Ledger9Bridge { &mut self, state_key: PassFatPointerAndRead<&[u8]>, ) -> AllocateAndReturnByCodec, LedgerApiError>> { + serve_pre_migration_v8_read!(*self, state_key, get_ledger_state_root(state_key)); + if is_unified(*self) { Bridge::::get_ledger_state_root(state_key) } else { @@ -528,3 +608,36 @@ pub trait Ledger9Bridge { }); } } + +#[cfg(all(test, feature = "std"))] +mod tests { + use super::as_ledger_9_error; + use crate::{ledger_8::types as v8, ledger_9::types as v9}; + + /// `as_ledger_9_error` relies on the two versions' `LedgerApiError` sharing a + /// SCALE encoding, which holds because both are generated from + /// `versions/common/types.rs`. Pin that down — including a nested payload and + /// the last variant, which is where a divergence would first show up — so a + /// future edit to one version's enum fails here rather than silently turning + /// every pre-migration read error into `HostApiError`. + #[test] + fn ledger_8_error_encoding_matches_ledger_9() { + let cases = [ + (v8::LedgerApiError::NoLedgerState, v9::LedgerApiError::NoLedgerState), + (v8::LedgerApiError::ContractNotPresent, v9::LedgerApiError::ContractNotPresent), + (v8::LedgerApiError::BeneficiaryNotFound, v9::LedgerApiError::BeneficiaryNotFound), + ( + v8::LedgerApiError::Deserialization(v8::DeserializationError::TypedArenaKey), + v9::LedgerApiError::Deserialization(v9::DeserializationError::TypedArenaKey), + ), + ( + v8::LedgerApiError::Serialization(v8::SerializationError::LedgerParameters), + v9::LedgerApiError::Serialization(v9::SerializationError::LedgerParameters), + ), + ]; + + for (from, expected) in cases { + assert_eq!(as_ledger_9_error(from.clone()), expected, "mistranslated {from:?}"); + } + } +} diff --git a/ledger/src/lib.rs b/ledger/src/lib.rs index 5517115b1..1ad985a36 100644 --- a/ledger/src/lib.rs +++ b/ledger/src/lib.rs @@ -194,6 +194,22 @@ pub fn init_ledger_storage_unified< } } +/// Returns true if `state_key` is a ledger-8 arena root, i.e. a tagged-serialized +/// `TypedArenaKey, _>`. +#[cfg(feature = "std")] +pub(crate) fn is_ledger_8_state_key(state_key: &[u8]) -> bool { + use ledger_storage_ledger_8::{DefaultDB, arena::TypedArenaKey, db::DB}; + use midnight_serialize::Tagged; + + type Ledger8Root = TypedArenaKey, ::Hasher>; + + let expected = ::tag(); + match midnight_serialize::peek_tag(&mut std::io::Cursor::new(state_key)) { + Ok(tag) => tag.as_str() == expected.as_ref(), + Err(_) => false, + } +} + mod common; pub mod types { @@ -237,4 +253,32 @@ mod tests { unsafe_drop_default_storage::(); assert!(try_get_default_storage::().is_none()); } + + /// `is_ledger_8_state_key` is what the ledger-9 host API dispatches on to read the + /// `set_code` block of the 8->9 hardfork, whose `StateKey` is one version behind + /// its `:code` (GH #1959). It has to tell a ledger-8 arena root from a ledger-9 + /// one from the header tag alone. + #[test] + fn ledger_8_state_key_tag_is_recognised() { + use ledger_storage_ledger_8::DefaultDB; + use midnight_serialize::{GLOBAL_TAG, Tagged}; + + // A `StateKey` is `tagged_serialize(&Sp, D>::as_typed_key())`, and + // `TypedArenaKey`'s tag wraps its referent's — which for `Ledger` is just + // `LedgerState`'s. Only the header matters here; `peek_tag` never reads the body. + fn header() -> Vec { + format!("{GLOBAL_TAG}storage-key({}):", T::tag()).into_bytes() + } + let v8 = header::>(); + let v9 = header::>(); + assert_ne!(v8, v9, "v8 and v9 ledger states must not share a tag"); + + assert!(super::is_ledger_8_state_key(&v8)); + assert!(!super::is_ledger_8_state_key(&v9)); + + // An unset `StateKey`, or anything else untagged, is not a ledger-8 root: the + // host API must take its ordinary ledger-9 path rather than guess. + assert!(!super::is_ledger_8_state_key(&[])); + assert!(!super::is_ledger_8_state_key(b"not-tagged-at-all")); + } } diff --git a/util/toolkit/tests/hardfork_e2e.rs b/util/toolkit/tests/hardfork_e2e.rs index 8507a90ac..da7d11a63 100644 --- a/util/toolkit/tests/hardfork_e2e.rs +++ b/util/toolkit/tests/hardfork_e2e.rs @@ -19,6 +19,7 @@ use clap::Parser; use common::{test_image, wait_for_node::wait_for_finalized_block}; use midnight_node_toolkit::cli::{Cli, run_command}; use std::{process::Command, time::Duration}; +use subxt::rpcs::{RpcClient, rpc_params}; use testcontainers::{ GenericImage, ImageExt, core::{ContainerPort, WaitFor}, @@ -53,6 +54,111 @@ async fn run_cli(args: &[&str]) { eprintln!("[hardfork_e2e] CLI command succeeded"); } +/// Hash of the block at `height`, hex-encoded. +async fn block_hash_at(rpc: &RpcClient, height: u64) -> String { + let hash: serde_json::Value = rpc + .request("chain_getBlockHash", rpc_params![height]) + .await + .unwrap_or_else(|e| panic!("chain_getBlockHash({height}) failed: {e}")); + hash.as_str() + .unwrap_or_else(|| panic!("no block at height {height}")) + .to_owned() +} + +/// The runtime `specVersion` *stored at* `hash`. +/// +/// Raw `state_getRuntimeVersion` rather than subxt's typed metadata on purpose: +/// across a runtime upgrade the client's metadata follows the new runtime, so +/// anything the old runtime encoded decodes unreliably. See +/// `midnight_node_toolkit::commands::runtime_upgrade`. +async fn spec_version_at(rpc: &RpcClient, hash: &str) -> u64 { + let version: serde_json::Value = rpc + .request("state_getRuntimeVersion", rpc_params![hash]) + .await + .unwrap_or_else(|e| panic!("state_getRuntimeVersion({hash}) failed: {e}")); + version + .get("specVersion") + .and_then(|v| v.as_u64()) + .unwrap_or_else(|| panic!("no specVersion in runtime version at {hash}")) +} + +async fn finalized_height(rpc: &RpcClient) -> u64 { + let hash: serde_json::Value = rpc + .request("chain_getFinalizedHead", rpc_params![]) + .await + .expect("chain_getFinalizedHead failed"); + let header: serde_json::Value = rpc + .request("chain_getHeader", rpc_params![hash]) + .await + .expect("chain_getHeader failed"); + header + .get("number") + .and_then(|n| n.as_str()) + .and_then(|s| u64::from_str_radix(s.trim_start_matches("0x"), 16).ok()) + .expect("no number in finalized header") +} + +/// Locate the block that applied the new runtime code — the one whose committed +/// state pairs the *new* `:code` with the *old* ledger version's `StateKey`. +/// +/// `frame_system` overwrites `:code` inside that block (the pre-fork runtime ships +/// `system_version: 1`, so the code is not staged in `:pending_code`), while +/// pallet-midnight's v8->v9 state translation only runs in the next block's +/// `initialize_block`. Executing a read at that hash therefore runs ledger-9 WASM +/// against a ledger-8 arena root, which is what GH #1959 reports. +/// +/// `state_getRuntimeVersion` reports the code stored at a block, so the first +/// height reporting the new spec is exactly that block. spec_version is monotonic +/// along the chain, so binary-search for it. +async fn find_code_applied_block(rpc: &RpcClient, head: u64, old_spec: u64) -> u64 { + let (mut lo, mut hi) = (1u64, head); + while lo < hi { + let mid = lo + (hi - lo) / 2; + let hash = block_hash_at(rpc, mid).await; + if spec_version_at(rpc, &hash).await > old_spec { + hi = mid; + } else { + lo = mid + 1; + } + } + lo +} + +/// Every way of reading the ledger state must answer at `height`. +/// +/// Both the `midnight_*` RPCs and a raw `state_call`: the fix lives in the ledger-9 +/// host function, so the runtime API has to work at the skew block too — that is +/// the path subxt-based tooling (`chain-indexer`, GH #1969) takes, and it does not +/// go anywhere near the node's own RPC layer. +async fn assert_ledger_state_readable(rpc: &RpcClient, height: u64, label: &str) { + let hash = block_hash_at(rpc, height).await; + + for method in ["midnight_zswapStateRoot", "midnight_ledgerStateRoot"] { + let root: Vec = rpc + .request(method, rpc_params![&hash]) + .await + .unwrap_or_else(|e| panic!("{method} failed at {label} (#{height}, {hash}): {e}")); + assert!(!root.is_empty(), "{method} returned an empty root at {label} (#{height})"); + } + + // `Result, LedgerApiError>` SCALE-encoded: a leading 0x00 is `Ok`, and + // anything else is the pallet reporting a ledger error (0x01 plus the variant). + for api in + ["MidnightRuntimeApi_get_ledger_state_root", "MidnightRuntimeApi_get_ledger_parameters"] + { + let encoded: String = rpc + .request("state_call", rpc_params![api, "0x", &hash]) + .await + .unwrap_or_else(|e| panic!("{api} failed at {label} (#{height}, {hash}): {e}")); + assert!( + encoded.starts_with("0x00"), + "{api} returned an error at {label} (#{height}): {encoded}" + ); + } + + eprintln!("[hardfork_e2e] ledger state readable at {label} (#{height})"); +} + #[test_log::test(tokio::test)] async fn hardfork_single_tx() { // 1. Generate chain-spec from fork-from node @@ -140,7 +246,34 @@ async fn hardfork_single_tx() { ]) .await; - // 5. Post-fork: run single-tx again to verify the node still works after the (future) upgrade + // 5. GH #1959: the whole fork boundary must stay readable. The block that + // applied the new code carries a ledger-8 `StateKey` under ledger-9 `:code`, + // so the ledger-9 host API has to detect that and serve the read from the + // ledger-8 bridge; the block after it is already translated to v9 and must + // keep taking the ordinary ledger-9 path. + let rpc = RpcClient::from_insecure_url(&url).await.expect("failed to open raw RPC client"); + let pre_fork_spec = { + let hash = block_hash_at(&rpc, 1).await; + spec_version_at(&rpc, &hash).await + }; + let head = finalized_height(&rpc).await; + let applied = find_code_applied_block(&rpc, head, pre_fork_spec).await; + eprintln!( + "[hardfork_e2e] new runtime code applied at #{applied} \ + (pre-fork spec {pre_fork_spec}, finalized head #{head})" + ); + assert!(applied > 1, "expected the code-applying block to be past #1, got #{applied}"); + assert!(applied < head, "expected the code-applying block to be below the finalized head"); + + // `runtime-upgrade` already waits for finality to pass `applied`, but the + // assertion below needs `applied + 1` to exist regardless. + wait_for_finalized_block(&url, applied + 1, Duration::from_secs(60)).await; + + assert_ledger_state_readable(&rpc, applied - 1, "pre-fork").await; + assert_ledger_state_readable(&rpc, applied, "code-applied block").await; + assert_ledger_state_readable(&rpc, applied + 1, "post-migration").await; + + // 6. Post-fork: run single-tx again to verify the node still works after the (future) upgrade run_cli(&[ "generate-txs", "--fetch-cache", From a5310e4685eba57d14f577b487e3e625ea8a45be Mon Sep 17 00:00:00 2001 From: Oscar Bailey <79094698+ozgb@users.noreply.github.com> Date: Wed, 12 Aug 2026 12:11:25 +0100 Subject: [PATCH 05/13] feat: reset Dust state and re-apply cNight UTxOs during migration (#2012) Squashed backport of the ozgb-cnight-mbm branch (PR #2012), which is still open on main. The ledger 8->9 state translation wipes the ledger's dust state; this rebuilds cNIGHT's slice of the dust generating set as a multi-block migration driven from `UtxoOwners` plus the retained v8 arena root, backdating each replayed ctime to the DUST cap. Squashed rather than cherry-picked commit-by-commit because the branch contains a merge from main (16f07a84e) that would drag in unrelated changes. PR: https://github.com/midnightntwrk/midnight-node/pull/2012 Assisted-by: Claude:claude-opus-5 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> --- .../added/cnight-dust-generation-reapply.md | 61 +++ ledger/helpers/src/fork/fork_8_to_9.rs | 8 +- .../helpers/src/state_translation_v8_to_v9.rs | 29 +- .../src/versions/common/wallet/dust.rs | 47 +++ ledger/src/host_api/dust_generation.rs | 234 +++++++++++ ledger/src/host_api/ledger_9.rs | 31 ++ ledger/src/host_api/mod.rs | 5 + pallets/cnight-observation/mock/src/mock.rs | 2 + .../mock/src/mock_with_capture.rs | 34 +- pallets/cnight-observation/src/lib.rs | 80 +++- pallets/cnight-observation/src/migrations.rs | 1 + .../cnight-observation/src/migrations/v2.rs | 364 ++++++++++++++++++ .../tests/dust_reapply_tests.rs | 338 ++++++++++++++++ .../tests/migration_tests.rs | 101 ++++- pallets/cnight-observation/tests/tests.rs | 43 ++- pallets/midnight/src/lib.rs | 6 +- primitives/midnight/src/lib.rs | 9 +- runtime/src/lib.rs | 13 +- util/toolkit/tests/hardfork_e2e.rs | 76 +++- 19 files changed, 1445 insertions(+), 37 deletions(-) create mode 100644 changes/runtime/added/cnight-dust-generation-reapply.md create mode 100644 ledger/src/host_api/dust_generation.rs create mode 100644 pallets/cnight-observation/src/migrations/v2.rs create mode 100644 pallets/cnight-observation/tests/dust_reapply_tests.rs diff --git a/changes/runtime/added/cnight-dust-generation-reapply.md b/changes/runtime/added/cnight-dust-generation-reapply.md new file mode 100644 index 000000000..b75a46959 --- /dev/null +++ b/changes/runtime/added/cnight-dust-generation-reapply.md @@ -0,0 +1,61 @@ +#cnight #dust #migration +# Re-apply cNIGHT dust generation after the ledger 8 -> 9 hardfork + +The ledger 8 -> 9 hardfork wipes dust state, which would silently stop DUST +generation for every cNIGHT holder. `pallet-cnight-observation` now rebuilds its +own slice of the ledger's dust generating set as a multi-block migration +(storage version 1 -> 2): + +- a single-block migration run in the upgrade block saves the pre-fork ledger-8 + arena root, the only place the wiped entries' night value and dust owner + survive; +- the multi-block migration then pages through `UtxoOwners` (the set of nonces + that are cnight's and still live), reads each nonce's pre-wipe value and owner + through a new `dust_generation_values_v8` host function (which also serves the + state's dust `time_to_cap`), and re-applies them in batches of 200 as + `CNightGeneratesDustUpdate` system transactions; +- incoming Cardano observations are ignored while it runs — the existing storage + version gate in `process_tokens` covers it, so `NextCardanoPosition` does not + advance and the observer re-delivers the same UTXOs afterwards. + +It is paced at one batch per block on purpose. `process_tokens`' benchmark +observes registration UTXOs, which never reach the ledger, so its weight says +nothing about the cost of 200 dust `Create`s; left to the weight meter the MBM +service budget would service ~100 batches — 20k ledger creates — in one block. +Measured live sets on 2026-08-06: **mainnet 4870** nonces (finalized #2019697), +**preview 1524** (#301994), **preprod 85** (#1985972) — so ~25 blocks (~2.5 min) +of gated observation on mainnet, ~8 on preview, 1 on preprod. + +The restored generation entries are field-for-field identical to the wiped ones. +Only the accrual clock moves: the original creation time lives on the dust UTXO +the wipe takes, so the replay stamps `fork block time - dust.time_to_cap()` +(~1 week). DUST accrues linearly from the creation time to a cap of +`night_value * night_dust_ratio` reached after `time_to_cap`, so backdating by +exactly that much puts every holder at their cap the moment the replay lands — +the pre-fork steady state, since anyone who had held cNIGHT for a week was +already capped. Stamping the fork block itself would instead start everyone at +zero and refill over a week in proportion to holdings, locking small holders out +of paying fees for days. The real per-UTXO creation time is only available from +db-sync, and would restore holders to the same cap anyway for all but the +youngest UTXOs, at the price of a new consensus-critical mainchain query in the +middle of the hardfork; the chosen offset over-credits only cNIGHT locked in the +last week, bounded by a cap it would have reached regardless. + +**Only cnight's slice of the generating set is restored.** Native NIGHT registers +generation entries too (at dust registration, for delegated unshielded outputs, +and on the mint/claim path); nothing in this repo records which of those the wipe +took, so `DustReapplyCompleted` must not be read as "all dust generation +restored". + +The wipe itself is part of this change: the v8 -> v9 state translation now drops +the dust state and installs the empty one genesis starts from, instead of +carrying it across. The toolkit's `fork_context_8_to_9` mirrors that, resetting +every wallet's local dust state so it does not try to spend dust the chain no +longer has. Should a translation ever carry dust across again, the migration +self-cancels rather than corrupting state: the first replayed event collides with +`GenerationInfoAlreadyPresent` and it emits `DustReapplySkipped`. + +New events: `DustReapplyStarted`, `DustReapplyBatchFailed`, +`DustReapplyCompleted`, `DustReapplySkipped`. + +PR: diff --git a/ledger/helpers/src/fork/fork_8_to_9.rs b/ledger/helpers/src/fork/fork_8_to_9.rs index fac03ec49..540323ebc 100644 --- a/ledger/helpers/src/fork/fork_8_to_9.rs +++ b/ledger/helpers/src/fork/fork_8_to_9.rs @@ -114,7 +114,7 @@ pub fn fork_context_8_to_9( }) }) .transpose(); - let new_wallet = Wallet { + let mut new_wallet = Wallet { root_seed: v.root_seed.as_ref().map(|s| { WalletSeed::try_from(s.as_bytes()) .expect("wallet seed different length between versions") @@ -135,6 +135,12 @@ pub fn fork_context_8_to_9( dust: (*old_to_new_sp::<_, DustWallet>(crate::ledger_8::Sp::new(v.dust.clone()))?) .clone(), }; + // The fork wipes the on-chain dust state (see + // `state_translation_v8_to_v9`), so the wallet's view of its dust — UTxOs, + // generation info, merkle witnesses — is stale the moment we cross it. + // Reset it to a freshly-derived state under the v9 dust parameters; dust + // re-accrues from the post-fork generation entries the chain replays. + new_wallet.dust.wipe_local_state(&ledger_state.parameters); let new_key: WalletSeed = old_to_new_ser_untagged(&k)?; wallets.insert(new_key, new_wallet); } diff --git a/ledger/helpers/src/state_translation_v8_to_v9.rs b/ledger/helpers/src/state_translation_v8_to_v9.rs index 4106b7819..7b9f9ac7e 100644 --- a/ledger/helpers/src/state_translation_v8_to_v9.rs +++ b/ledger/helpers/src/state_translation_v8_to_v9.rs @@ -32,8 +32,10 @@ //! | ContractOperation | `contract-operation[v4]` | `contract-operation[v6]` | single `v2` key -> `{ v2, v3, ir }`; v8 key maps to `v2`, new `v3`/`ir` empty | //! | ContractMaintenanceAuthority | `contract-maintenance-authority[v1]` | `contract-maintenance-authority[v2]` | `committee: Vec` -> `Vec` (Schnorr/ECDSA sum) | //! -//! Everything else (zswap, utxo, dust, replay_protection, treasury, -//! unclaimed_block_rewards) is tag-stable and passes through `recast`. +//! Everything else (zswap, utxo, replay_protection, treasury, +//! unclaimed_block_rewards) is tag-stable and passes through `recast`. `dust` +//! is the exception: it is tag-stable but deliberately *wiped* rather than +//! carried over (see [`LedgerStateTl::finalize`]). // Map the upstream translation crate names onto the node workspace's package // aliases. `mn-ledger-8`/`mn-ledger-9` are the two `midnight-ledger` majors; @@ -311,7 +313,13 @@ impl contract: Map { mpt: contract_mpt.force_downcast(), key_type: PhantomData }, utxo: recast(&source.utxo)?, replay_protection: recast(&source.replay_protection)?, - dust: recast(&source.dust)?, + // The hardfork wipes dust: the v8 dust state is dropped and replaced + // with the same empty state genesis starts from. Dust generation for + // still-locked cNIGHT is re-applied afterwards by + // `pallet_cnight_observation::migrations::v2`; dust UTxOs (balances) + // are not restored — they regenerate from the re-applied generation + // entries. + dust: Sp::new(ledger_v9::dust::DustState::default()), })) } } @@ -707,4 +715,19 @@ mod tests { serialize::tagged_deserialize(&mut &buf[..]).expect("v9 deserialize"); assert_eq!(v9_rt.network_id, v9.network_id); } + + /// The translation wipes dust: whatever generation/utxo state v8 held, the + /// v9 side comes out as the empty state genesis starts from. + #[test] + fn dust_state_is_wiped() { + let mut v8 = ledger_v8::structure::LedgerState::::new("test-network"); + let mut dust = (*v8.dust).clone(); + dust.generation.generating_tree_first_free = 7; + dust.utxo.commitments_first_free = 3; + v8.dust = Sp::new(dust); + + let v9 = translate_to_completion(v8); + + assert_eq!(*v9.dust, ledger_v9::dust::DustState::default()); + } } diff --git a/ledger/helpers/src/versions/common/wallet/dust.rs b/ledger/helpers/src/versions/common/wallet/dust.rs index f05461afa..758727552 100644 --- a/ledger/helpers/src/versions/common/wallet/dust.rs +++ b/ledger/helpers/src/versions/common/wallet/dust.rs @@ -69,6 +69,21 @@ impl DustWallet { Ok(Self::from_seed(derived_seed, params)) } + /// Drop everything this wallet knows about its dust — UTxOs, generation + /// info, merkle witnesses, in-flight spends — keeping only its keys, as if + /// it had just been derived. Used to mirror a hardfork that wipes the + /// on-chain dust state (ledger 8 -> 9): keeping the old local state would + /// have the wallet spend dust the chain no longer has. + /// + /// A watch-only wallet (no local state) stays watch-only. + pub fn wipe_local_state(&mut self, params: &LedgerParameters) { + self.dust_local_state = self + .dust_local_state + .as_ref() + .map(|_| Sp::new(DustLocalState::new(params.dust))); + self.spent_utxos = HashSet::new(); + } + pub fn replay_events<'a>( &mut self, events: impl IntoIterator>, @@ -218,6 +233,38 @@ mod tests { let path = DerivationPath::default_for_role(Role::UnshieldedExternal); assert!(DustWallet::::from_path(test_seed(), &path, None).is_err()); } + + #[test] + fn wipe_local_state_resets_state_but_keeps_keys() { + let params = super::super::super::INITIAL_PARAMETERS; + let mut wallet = DustWallet::::default(test_seed(), Some(¶ms)); + let public_key = wallet.public_key; + + // Stand in for a synced-up wallet: a non-default local state, as it would + // look after replaying pre-fork dust events. + let mut synced = (**wallet.dust_local_state.as_ref().unwrap()).clone(); + synced.sync_time = super::Timestamp::from_secs(1_000); + wallet.dust_local_state = Some(super::Sp::new(synced)); + + wallet.wipe_local_state(¶ms); + + let wiped = wallet.dust_local_state.as_ref().unwrap(); + assert_eq!(wiped.sync_time, super::Timestamp::default()); + assert_eq!(wallet.public_key, public_key, "keys must survive the wipe"); + assert!( + wallet + .speculative_spend(1, super::Timestamp::from_secs(1_000), ¶ms.dust) + .is_ok() + ); + } + + /// A watch-only wallet (address-derived, no local state) must stay that way. + #[test] + fn wipe_local_state_leaves_watch_only_wallets_alone() { + let mut wallet = DustWallet::::default(test_seed(), None); + wallet.wipe_local_state(&super::super::super::INITIAL_PARAMETERS); + assert!(wallet.dust_local_state.is_none()); + } } #[derive(Debug, Error)] diff --git a/ledger/src/host_api/dust_generation.rs b/ledger/src/host_api/dust_generation.rs new file mode 100644 index 000000000..8a367c64f --- /dev/null +++ b/ledger/src/host_api/dust_generation.rs @@ -0,0 +1,234 @@ +// This file is part of midnight-node. +// Copyright (C) 2025-2026 Midnight Foundation +// SPDX-License-Identifier: Apache-2.0 +// Licensed under the Apache License, Version 2.0 (the "License"); +// You may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Host-side batched read of the *pre-fork* (ledger-8) dust generation state. +//! +//! The ledger 8 -> 9 hardfork wipes dust state, so `pallet-cnight-observation` +//! has to re-apply the cNIGHT-generates-DUST entries it fed the ledger over the +//! chain's life (see `pallet_cnight_observation::migrations::v2`). Its own +//! storage records only which nonces are cnight's (`UtxoOwners`) — the night +//! `value` and the dust `owner` of each entry live in the about-to-be-wiped +//! `DustGenerationInfo`, which is what this module reads back out. +//! +//! It reads the *v8* state through the arena root the migration saved before +//! translation (`pallet_cnight_observation::PreForkStateKey`), which stays +//! resolvable because the arena retains historical ledger states. Reading a v8 +//! state from the v9 bridge is the same trick `serve_pre_migration_v8_read!` +//! plays in [`crate::host_api::ledger_9`]. + +use crate::ledger_8::api::Ledger as Ledger8; +use crate::ledger_9::types::{DeserializationError, LedgerApiError}; + +use base_crypto::{hash::HashOutput, time::Timestamp}; +use ledger_storage_ledger_8 as storage; +use midnight_serialize::{Serializable, tagged_deserialize}; +use mn_ledger_8::dust::InitialNonce; +use storage::{ + arena::{Sp, TypedArenaKey}, + db::DB, + storage::default_storage, +}; + +const LOG_TARGET: &str = "midnight::ledger::dust_generation"; + +/// The dust parameters' `time_to_cap` in seconds, plus — for each requested +/// initial nonce — the night value and dust owner of its still-generating entry +/// in the ledger-8 dust state referenced by `state_key`; `None` when the nonce +/// is not tracked, or has already been destroyed. Positionally aligned with +/// `nonces`. +/// +/// `time_to_cap` is how far the caller backdates the replayed `ctime` so every +/// restored entry lands at its DUST cap, i.e. at the balance it held before the +/// wipe. It comes from the v8 state because that is the one already loaded here, +/// and the 8 -> 9 translation recasts `parameters.dust` unchanged. +/// +/// The owner bytes are the (untagged) serialized `DustPublicKey`, i.e. exactly +/// what `construct_cnight_generates_dust_event` accepts for `owner`. +/// +/// Errors with `NoLedgerState` when `state_key` is not a ledger-8 state. This +/// must *not* be an all-`None` success: the caller's key is only v8 because its +/// `RecordPreForkState` migration runs before the pallet-midnight translation, +/// so a future reorder of the runtime `Migrations` tuple would otherwise +/// silently restore nothing, chain-wide, detectable only by absent DUST. +pub fn dust_generation_values_v8( + state_key: &[u8], + nonces: &[[u8; 32]], +) -> Result<(u64, Vec)>>), LedgerApiError> { + if !crate::is_ledger_8_state_key(state_key) { + log::error!( + target: LOG_TARGET, + "pre-fork state key is not a ledger-8 arena root; refusing to serve dust generation values" + ); + return Err(LedgerApiError::NoLedgerState); + } + + let key8: TypedArenaKey, D::Hasher> = tagged_deserialize(&mut &state_key[..]) + .map_err(|e| { + log::error!(target: LOG_TARGET, "failed to deserialize v8 state key: {e:?}"); + LedgerApiError::Deserialization(DeserializationError::TypedArenaKey) + })?; + // One arena load, amortised over the whole batch. + let ledger8: Sp, D> = default_storage::().arena.get_lazy(&key8).map_err(|e| { + log::error!(target: LOG_TARGET, "failed to load v8 ledger from arena: {e:?}"); + LedgerApiError::NoLedgerState + })?; + let generation = &ledger8.state.dust.generation; + // Non-negative by construction (`night_dust_ratio / generation_decay_rate`). + let time_to_cap = ledger8.state.parameters.dust.time_to_cap().as_seconds().max(0) as u64; + + let values = nonces + .iter() + .map(|nonce| { + // Same lookup path the ledger's own `Destroy` handler takes: + // nonce -> leaf index -> generating tree leaf. + let Some(index) = generation.night_indices.get(&InitialNonce(HashOutput(*nonce))) else { + // The caller only asks about nonces it believes are live, so an + // untracked one is a pallet/ledger divergence, same as a destroyed + // one below. + log::warn!(target: LOG_TARGET, "nonce {} is not tracked in the v8 dust state", hex::encode(nonce)); + return None; + }; + let Some((_, info)) = generation.generating_tree.index(*index) else { + log::error!( + target: LOG_TARGET, + "invariant violated: `night_indices` entry for {} not backed in `generating_tree`", + hex::encode(nonce), + ); + return None; + }; + // A destroyed entry keeps its leaf forever, with `dtime` rewritten. + // The caller's `UtxoOwners` is meant to be exactly the live set, so + // this is a pallet/ledger divergence worth surfacing. + if info.dtime != Timestamp::MAX { + log::warn!( + target: LOG_TARGET, + "nonce {} is already destroyed in the v8 dust state (dtime {:?}); not restoring", + hex::encode(nonce), + info.dtime, + ); + return None; + } + let mut owner = Vec::new(); + if let Err(e) = Serializable::serialize(&info.owner, &mut owner) { + log::error!(target: LOG_TARGET, "failed to serialize dust owner: {e:?}"); + return None; + } + Some((info.value, owner)) + }) + .collect(); + + Ok((time_to_cap, values)) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::ledger_9::api::Ledger as Ledger9; + use ledger_storage_ledger_8::db::InMemoryDB; + use midnight_serialize::tagged_serialize; + use mn_ledger_8::{ + dust::DustPublicKey, + structure::{ + CNightGeneratesDustActionType, CNightGeneratesDustEvent, LedgerState, SystemTransaction, + }, + }; + use transient_crypto::curve::Fr; + + const NONCE_LIVE: [u8; 32] = [1; 32]; + const NONCE_DESTROYED: [u8; 32] = [2; 32]; + const NONCE_UNKNOWN: [u8; 32] = [3; 32]; + + fn event( + nonce: [u8; 32], + value: u128, + owner: DustPublicKey, + action: CNightGeneratesDustActionType, + time_secs: u64, + ) -> CNightGeneratesDustEvent { + CNightGeneratesDustEvent { + value, + owner, + time: Timestamp::from_secs(time_secs), + action, + nonce: InitialNonce(HashOutput(nonce)), + } + } + + /// Persist `ledger` into the default in-memory arena and return its root, in + /// the same `StateKey` shape the pallet stores. + fn root_of + midnight_serialize::Tagged>(value: T) -> Vec { + let mut sp = default_storage::().arena.alloc(value); + sp.persist(); + let mut bytes = Vec::new(); + tagged_serialize(&sp.as_typed_key(), &mut bytes).expect("serialize root"); + bytes + } + + /// A v8 dust state holding a live and a destroyed cnight entry. + fn v8_root(owner: DustPublicKey) -> Vec { + use CNightGeneratesDustActionType::*; + let state = LedgerState::::new("local-test"); + let (state, _) = state + .apply_system_tx( + &SystemTransaction::CNightGeneratesDustUpdate { + events: vec![ + event(NONCE_LIVE, 100, owner, Create, 1_000), + event(NONCE_DESTROYED, 200, owner, Create, 1_000), + event(NONCE_DESTROYED, 200, owner, Destroy, 2_000), + ], + }, + Timestamp::from_secs(2_000), + ) + .expect("apply v8 cnight dust update"); + root_of(Ledger8::new(state)) + } + + #[test] + fn serves_live_entries_and_skips_destroyed_and_unknown() { + let owner = DustPublicKey(Fr::from(7u64)); + let root = v8_root(owner); + + let (time_to_cap, values) = dust_generation_values_v8::( + &root, + &[NONCE_LIVE, NONCE_DESTROYED, NONCE_UNKNOWN], + ) + .expect("v8 root must resolve"); + + let mut expected_owner = Vec::new(); + Serializable::serialize(&owner, &mut expected_owner).unwrap(); + + assert_eq!(values, vec![Some((100u128, expected_owner)), None, None]); + assert_eq!( + time_to_cap, + mn_ledger_8::structure::INITIAL_PARAMETERS.dust.time_to_cap().as_seconds() as u64, + "the served cap offset must be the state's own dust parameter", + ); + } + + /// The migration only ever holds a v8 key by construction, so a v9 key means + /// the migration order changed underneath us — that must fail loudly rather + /// than restore nothing. + #[test] + fn v9_state_key_is_rejected() { + let root = root_of(Ledger9::new(mn_ledger_9::structure::LedgerState::::new( + "local-test", + ))); + + assert!(matches!( + dust_generation_values_v8::(&root, &[NONCE_LIVE]), + Err(LedgerApiError::NoLedgerState), + )); + } +} diff --git a/ledger/src/host_api/ledger_9.rs b/ledger/src/host_api/ledger_9.rs index 982082939..e696fdacd 100644 --- a/ledger/src/host_api/ledger_9.rs +++ b/ledger/src/host_api/ledger_9.rs @@ -575,6 +575,37 @@ pub trait Ledger9Bridge { } } + /// The dust `time_to_cap` in seconds, plus the night value and dust owner of + /// each requested nonce's still-generating entry in the *pre-fork* + /// (ledger-8) dust state, positionally aligned with `nonces` and `None` for + /// nonces that are untracked or already destroyed. + /// + /// Called by `pallet-cnight-observation`'s dust re-apply migration, which + /// rebuilds the cNIGHT generation entries the ledger 8 -> 9 hardfork wipes + /// and backdates their `ctime` by `time_to_cap`. `state_key` is the v8 arena + /// root it saved during the upgrade block; `Err(NoLedgerState)` means that + /// root no longer resolves, or is not a ledger-8 root at all. + fn dust_generation_values_v8( + &mut self, + state_key: PassFatPointerAndRead<&[u8]>, + nonces: PassFatPointerAndDecode>, + ) -> AllocateAndReturnByCodec)>>), LedgerApiError>> { + // The migration runs in `inherents_applied()`, after pallet-midnight has + // initialized the arena, but `set_default_storage` is idempotent and + // keeps this callable from anywhere in the block. + if is_unified(*self) { + Bridge::::set_default_storage(*self); + crate::host_api::dust_generation::dust_generation_values_v8::( + state_key, &nonces, + ) + } else { + Bridge::::set_default_storage(*self); + crate::host_api::dust_generation::dust_generation_values_v8::( + state_key, &nonces, + ) + } + } + /// Initialize a process-wide temporary ledger ParityDb seeded with the /// undeployed-network genesis state. /// diff --git a/ledger/src/host_api/mod.rs b/ledger/src/host_api/mod.rs index 1b1fd44c3..778f942a3 100644 --- a/ledger/src/host_api/mod.rs +++ b/ledger/src/host_api/mod.rs @@ -18,3 +18,8 @@ pub mod ledger_9; /// Host-side v8 -> v9 ledger state translation used by the runtime storage migration. #[cfg(feature = "std")] pub mod migration_8_to_9; + +/// Host-side read of the pre-fork (ledger-8) dust generation state, used by the +/// cNIGHT dust re-apply migration. +#[cfg(feature = "std")] +pub mod dust_generation; diff --git a/pallets/cnight-observation/mock/src/mock.rs b/pallets/cnight-observation/mock/src/mock.rs index bd667521f..b063b0134 100644 --- a/pallets/cnight-observation/mock/src/mock.rs +++ b/pallets/cnight-observation/mock/src/mock.rs @@ -148,6 +148,8 @@ parameter_types! { impl pallet_cnight_observation::Config for Test { type MidnightSystemTransactionExecutor = MidnightSystem; + type LedgerStateProvider = Midnight; + type LedgerBlockContextProvider = Midnight; type WeightInfo = (); } diff --git a/pallets/cnight-observation/mock/src/mock_with_capture.rs b/pallets/cnight-observation/mock/src/mock_with_capture.rs index 7d6cfa39b..1c8e3ac52 100644 --- a/pallets/cnight-observation/mock/src/mock_with_capture.rs +++ b/pallets/cnight-observation/mock/src/mock_with_capture.rs @@ -20,7 +20,10 @@ use frame_support::sp_runtime::{ use frame_support::traits::{ConstU16, ConstU32, ConstU64}; use frame_support::weights::RuntimeDbWeight; use frame_support::*; -use midnight_primitives::MidnightSystemTransactionExecutor; +use midnight_node_ledger::latest::types::BlockContext; +use midnight_primitives::{ + LedgerBlockContextProvider, LedgerStateProvider, MidnightSystemTransactionExecutor, +}; use sidechain_domain::*; #[cfg(feature = "std")] use sp_io::TestExternalities; @@ -165,8 +168,37 @@ impl MidnightSystemTransactionExecutor for MidnightSystemTx { } } +parameter_types! { + /// Stand-ins for `pallet-midnight`, which this mock deliberately omits (it + /// exists to capture system transactions without a ledger). Tests drive the + /// dust replay migration through these. + pub static MockLedgerStateKey: Vec = Vec::new(); + pub static MockBlockTime: u64 = 1_700_000_000; +} + +pub struct MockLedger; + +impl LedgerStateProvider for MockLedger { + fn get_ledger_state_key() -> Vec { + MockLedgerStateKey::get() + } +} + +impl LedgerBlockContextProvider for MockLedger { + fn get_block_context() -> BlockContext { + BlockContext { + tblock: MockBlockTime::get(), + tblock_err: 0, + parent_block_hash: vec![0u8; 32], + last_block_time: 0, + } + } +} + impl pallet_cnight_observation::Config for Test { type MidnightSystemTransactionExecutor = MidnightSystemTx; + type LedgerStateProvider = MockLedger; + type LedgerBlockContextProvider = MockLedger; type WeightInfo = (); } diff --git a/pallets/cnight-observation/src/lib.rs b/pallets/cnight-observation/src/lib.rs index 7e40033f8..91788cd87 100644 --- a/pallets/cnight-observation/src/lib.rs +++ b/pallets/cnight-observation/src/lib.rs @@ -84,7 +84,9 @@ pub const MAX_UTXO_COUNT: u32 = DEFAULT_CARDANO_TX_CAPACITY_PER_BLOCK * UTXO_PER #[frame_support::pallet] pub mod pallet { use frame_support::sp_runtime::traits::Hash; - use midnight_primitives::MidnightSystemTransactionExecutor; + use midnight_primitives::{ + LedgerBlockContextProvider, LedgerStateProvider, MidnightSystemTransactionExecutor, + }; use midnight_primitives_cnight_observation::{ CARDANO_ASSET_NAME_MAX_LENGTH, CARDANO_BECH32_ADDRESS_MAX_LENGTH, CNIGHT_POLICY_ID_LENGTH, CardanoRewardAddressBytes, DustPublicKeyBytes, @@ -149,7 +151,9 @@ pub mod pallet { pub system_transaction_hash: LedgerHash, } - const STORAGE_VERSION: StorageVersion = StorageVersion::new(1); + // v2: re-apply the cNIGHT dust generation entries the ledger 8 -> 9 hardfork + // wipes (see `migrations::v2`). + const STORAGE_VERSION: StorageVersion = StorageVersion::new(2); #[pallet::pallet] #[pallet::storage_version(STORAGE_VERSION)] @@ -159,6 +163,11 @@ pub mod pallet { #[pallet::config] pub trait Config: frame_system::Config { type MidnightSystemTransactionExecutor: MidnightSystemTransactionExecutor; + /// Reads the ledger state key, to capture the pre-hardfork (ledger-8) + /// one before the pallet-midnight translation replaces it. + type LedgerStateProvider: LedgerStateProvider; + /// Supplies the ledger time stamped on the replayed dust events. + type LedgerBlockContextProvider: LedgerBlockContextProvider; /// Weight information for extrinsics in this pallet. type WeightInfo: crate::weights::WeightInfo; } @@ -171,6 +180,27 @@ pub mod pallet { MappingAdded(MappingEntry), MappingRemoved(MappingEntry), SystemTransactionApplied(SystemTransactionApplied), + /// The hardfork upgrade block armed the dust generation replay + /// (`migrations::v2`) by saving the pre-fork ledger state key. + DustReapplyStarted, + /// One replay batch failed to apply; its nonces were not restored. The + /// replay continues with the next batch. + DustReapplyBatchFailed { + nonces: Vec, + }, + /// The replay finished. `applied` entries were restored; `skipped` were + /// not (untracked, already destroyed, or in a failed batch). + /// + /// Note this covers cnight's slice of the ledger's dust generating set + /// only — native-NIGHT generation entries are not restored here. + DustReapplyCompleted { + applied: u32, + skipped: u32, + }, + /// The replay did not run: the hardfork did not wipe dust state, no + /// pre-fork state key was recorded, or that key is unreadable. The + /// reason is logged. + DustReapplySkipped, } #[pallet::error] @@ -294,6 +324,25 @@ pub mod pallet { #[pallet::storage] pub type InherentExecutedThisBlock = StorageValue<_, bool, ValueQuery>; + /// The ledger-8 arena root as of the hardfork upgrade block, retained so the + /// dust replay (`migrations::v2`) can read pre-wipe night values and owners + /// after `pallet_midnight::StateKey` has moved on to the v9 root. Mirrors + /// that item's shape. Killed when the replay finishes. + #[pallet::storage] + #[pallet::unbounded] + pub type PreForkStateKey = StorageValue<_, Vec, OptionQuery>; + + /// Ledger time stamped on every replayed dust event: the fork block's own + /// time, backdated by the dust `time_to_cap` so every restored entry lands + /// at its DUST cap rather than at zero. Written by the first replay step. + #[pallet::storage] + pub type DustReapplyCtime = StorageValue<_, u64, OptionQuery>; + + /// Running (applied, skipped) tallies of the dust replay — the only on-chain + /// evidence it ran to completion. Killed when the replay finishes. + #[pallet::storage] + pub type DustReapplyProgress = StorageValue<_, (u32, u32), ValueQuery>; + #[pallet::genesis_config] #[derive(frame_support::DefaultNoBound)] pub struct GenesisConfig { @@ -657,15 +706,24 @@ pub mod pallet { ensure!(!InherentExecutedThisBlock::::get(), Error::::InherentAlreadyExecuted); InherentExecutedThisBlock::::put(true); - // While a multi-block migration of `Mapping` is still draining v0 storage, - // `unique_dust_key` (and therefore `handle_registration`, - // `handle_registration_removal`, `handle_create`) reads only v1, missing - // any v0 row that hasn't been moved yet. Acting on that partial view would - // silently corrupt registration state — e.g. a deregistration whose v0 - // row is still pending would no-op here and then re-appear as live once - // the migration drains it. Skip processing entirely; `NextCardanoPosition` - // stays unchanged so the next block's inherent re-presents the same UTXOs - // (plus any new ones) and we resume once the migration finishes. + // Skip observation processing entirely while any multi-block migration of + // this pallet's storage is in flight; `NextCardanoPosition` stays + // unchanged so the next block's inherent re-presents the same UTXOs (plus + // any new ones) and we resume once the migration finishes. + // + // v0 -> v1 (`Mapping`): `unique_dust_key` (and therefore + // `handle_registration`, `handle_registration_removal`, `handle_create`) + // reads only v1, missing any v0 row not yet moved. Acting on that partial + // view would silently corrupt registration state — e.g. a deregistration + // whose v0 row is still pending would no-op here and then re-appear as + // live once the migration drains it. + // + // v1 -> v2 (dust generation replay): the replay re-applies `Create` + // events for every live `UtxoOwners` nonce. A concurrent spend would + // `take` a nonce the replay has not restored yet (its `Destroy` failing + // against a wiped ledger, then the nonce gone from the live set), and a + // concurrent create would race the replay's own system transaction. + if Pallet::::on_chain_storage_version() < STORAGE_VERSION { log::warn!( "cnight-observation: skipping process_tokens (on-chain storage version {:?} < {:?}); MBM in progress", diff --git a/pallets/cnight-observation/src/migrations.rs b/pallets/cnight-observation/src/migrations.rs index 5ae2e0980..5d8546f22 100644 --- a/pallets/cnight-observation/src/migrations.rs +++ b/pallets/cnight-observation/src/migrations.rs @@ -13,5 +13,6 @@ // limitations under the License. pub mod v1; +pub mod v2; pub const PALLET_MIGRATIONS_ID: &[u8; 25] = b"pallet-cnight-observation"; diff --git a/pallets/cnight-observation/src/migrations/v2.rs b/pallets/cnight-observation/src/migrations/v2.rs new file mode 100644 index 000000000..023c2523e --- /dev/null +++ b/pallets/cnight-observation/src/migrations/v2.rs @@ -0,0 +1,364 @@ +// Copyright (C) Midnight Foundation +// SPDX-License-Identifier: Apache-2.0 +// Licensed under the Apache License, Version 2.0 (the "License"); +// You may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Storage migration v1 → v2: re-apply cNIGHT dust generation after the +//! ledger 8 → 9 hardfork wipes dust state. +//! +//! Every cNIGHT UTXO this pallet observed fed a `Create` event into the ledger's +//! dust generating set. The hardfork wipes that state, so without this migration +//! every cNIGHT holder would silently stop generating DUST. Two parts: +//! +//! * [`RecordPreForkState`] — single-block, and **must run before** +//! `pallet_midnight::migrations::v2` in the runtime's `Migrations` tuple: it +//! saves the still-untranslated ledger-8 arena root, which is the only place +//! the wiped entries' night `value` and dust `owner` survive. +//! * [`MigrateV1ToV2`] — the multi-block replay. It pages through `UtxoOwners` +//! (read-only: the provenance-and-liveness filter for which nonces are +//! cnight's and still live), asks the host for each nonce's pre-wipe +//! `(value, owner)`, and applies one `CNightGeneratesDustUpdate` per step. +//! `process_tokens` is gated off for the duration by the storage version. +//! +//! The restored generation entries are field-for-field identical to the wiped +//! ones. Only the accrual clock moves: the original `ctime` is not publicly +//! visible in ledger state (it is stored as a commitment only), so the replay +//! stamps `fork block time - dust.time_to_cap()`. DUST accrues linearly from +//! `ctime` to a cap of `night_value * night_dust_ratio` reached after +//! `time_to_cap` (~1 week), so backdating by exactly that much puts every holder +//! at their cap the moment the replay lands — the pre-fork steady state, since +//! anyone holding cNIGHT for a week was already capped. +//! +//! Stamping the fork block itself instead would be an equally arbitrary clock +//! that starts everyone at zero and refills over a week, in proportion to +//! holdings: large holders recover in minutes, small ones are locked out of +//! paying fees for days. The real per-UTXO `ctime` is only available from +//! db-sync, and would restore holders to the same cap anyway for all but the +//! youngest UTXOs — while making the hardfork depend on a new +//! consensus-critical mainchain query. The chosen offset over-credits only +//! cNIGHT locked in the last week, bounded by a cap it would reach regardless. +//! +//! Only cnight's slice of the generating set is restored. Native NIGHT registers +//! generation entries too, and nothing in this repo records which of those the +//! wipe took. +//! +//! The wipe itself lives in the translation table +//! (`midnight_node_ledger_helpers::state_translation_v8_to_v9`), which replaces +//! the v8 dust state with the empty one. Should that ever stop being true, this +//! migration self-cancels rather than corrupting state: the first replayed +//! `Create` collides with `GenerationInfoAlreadyPresent` (see +//! [`MigrateV1ToV2::step`]). + +extern crate alloc; + +use alloc::vec::Vec; +use frame_support::{ + migrations::{MigrationId, SteppedMigration, SteppedMigrationError}, + pallet_prelude::*, + traits::OnRuntimeUpgrade, + weights::WeightMeter, +}; +use midnight_node_ledger::types::active_ledger_bridge as LedgerApi; +use midnight_primitives::{ + LedgerBlockContextProvider, LedgerStateProvider, MidnightSystemTransactionExecutor, +}; + +use super::PALLET_MIGRATIONS_ID; +use crate::{ + Config, DustReapplyCtime, DustReapplyProgress, Event, Pallet, PreForkStateKey, UtxoActionType, + UtxoOwners, +}; + +const LOG_TARGET: &str = "cnight-observation::migration"; + +/// Nonces restored per step, and hence per host call and per system transaction. +/// +/// Matches `DEFAULT_CARDANO_TX_CAPACITY_PER_BLOCK`: a batch this size is one +/// `CNightGeneratesDustUpdate` that `process_tokens` already applies in a single +/// block in production, so it is known to fit. It also bounds the blast radius +/// of a failed batch. +pub const MAX_REAPPLY_BATCH: u32 = 200; + +/// Saves the pre-hardfork ledger-8 arena root for [`MigrateV1ToV2`] to read the +/// wiped dust entries' values and owners from. +/// +/// Single-block and O(1). Must sit *before* `pallet_midnight::migrations::v2` in +/// the runtime `Migrations` tuple — that migration replaces +/// `pallet_midnight::StateKey` with the translated v9 root. +pub struct RecordPreForkState(core::marker::PhantomData); + +impl OnRuntimeUpgrade for RecordPreForkState { + fn on_runtime_upgrade() -> Weight { + let weight = T::DbWeight::get().reads_writes(2, 1); + + if Pallet::::on_chain_storage_version() >= 2 { + return weight; + } + if PreForkStateKey::::exists() { + // Should be impossible: `pallet_migrations` blocks `set_code` while + // an MBM is in flight, so the replay cannot still be holding a key. + log::error!( + target: LOG_TARGET, + "pre-fork ledger state key is already set; leaving it alone rather than overwriting" + ); + return weight; + } + + PreForkStateKey::::put(T::LedgerStateProvider::get_ledger_state_key()); + Pallet::::deposit_event(Event::::DustReapplyStarted); + log::info!(target: LOG_TARGET, "recorded pre-fork ledger state key for the dust generation replay"); + + weight + } +} + +/// Replays cnight's dust generation entries into the post-hardfork ledger state, +/// one `UtxoOwners` page per step. +pub struct MigrateV1ToV2(core::marker::PhantomData); + +impl SteppedMigration for MigrateV1ToV2 { + /// The last `UtxoOwners` nonce processed. + type Cursor = T::Hash; + type Identifier = MigrationId<25>; + + fn id() -> Self::Identifier { + MigrationId { pallet_id: *PALLET_MIGRATIONS_ID, version_from: 1, version_to: 2 } + } + + fn step( + cursor: Option, + meter: &mut WeightMeter, + ) -> Result, SteppedMigrationError> { + // One batch per step, and — by charging half a block — one step per block. + // + // The weight model cannot pace this: `process_tokens`' benchmark observes + // *registration* UTXOs, which never reach the ledger, so its ~15ms for 200 + // UTXOs says nothing about 200 dust `Create`s. Against + // `MbmServiceWeight` (80% of the block) that would service ~100 batches — + // 20k ledger dust creates — in a single block. Half a block is over the + // service budget for a second step and under it for the first, so exactly + // one batch lands per block, and never the fatal + // `required > MaxServiceWeight`. + // + // The cost is latency, and it is small: mainnet's live set was ~4.9k + // nonces on 2026-08-06 (preview ~1.5k, preprod ~85), i.e. ~25 batches, + // so ~25 blocks (~2.5 min) of gated observation. The observer re-delivers + // everything afterwards. + let required = Weight::from_parts(T::BlockWeights::get().max_block.ref_time() / 2, 0); + if meter.remaining().any_lt(required) { + return Err(SteppedMigrationError::InsufficientWeight { required }); + } + let _ = meter.try_consume(required); + + // Never return `Err` below this point: steps run under + // `FreezeChainOnFailedMigration`, so any error freezes the chain. Every + // failure path instead winds the replay up and lets the observer resume. + let Some(pre_fork_key) = PreForkStateKey::::get() else { + log::info!( + target: LOG_TARGET, + "no pre-fork ledger state key recorded; nothing to replay" + ); + return Ok(cancel::()); + }; + + // Read-only paging: `UtxoOwners` is not drained, it stays the live set. + let mut iter = match cursor { + Some(last) => UtxoOwners::::iter_from(UtxoOwners::::hashed_key_for(last)), + None => UtxoOwners::::iter(), + }; + let nonces: Vec = + iter.by_ref().take(MAX_REAPPLY_BATCH as usize).map(|(nonce, _)| nonce).collect(); + + let Some(last) = nonces.last().copied() else { + return Ok(complete::()); + }; + + let raw_nonces: Vec<[u8; 32]> = nonces.iter().map(|nonce| nonce.0).collect(); + let (time_to_cap, values) = + match LedgerApi::dust_generation_values_v8(&pre_fork_key, raw_nonces) { + Ok(values) => values, + Err(e) => { + // The pre-fork arena root has been reaped, or (defensively) is + // not a ledger-8 root at all. Nothing to restore from. + log::error!( + target: LOG_TARGET, + "pre-fork dust generation state is unreadable ({e:?}); abandoning the replay" + ); + return Ok(cancel::()); + }, + }; + + // Stamped once, on the first step that has something to restore, and + // reused by every later batch so the whole set shares one clock. Steps + // run in `inherents_applied()`, i.e. after the timestamp inherent, so + // `tblock` is the current block's own time; backdating it by + // `time_to_cap` puts every restored entry straight at its DUST cap. + let ctime = match DustReapplyCtime::::get() { + Some(ctime) => ctime, + None => { + let tblock = T::LedgerBlockContextProvider::get_block_context().tblock; + let ctime = tblock.saturating_sub(time_to_cap); + DustReapplyCtime::::put(ctime); + ctime + }, + }; + + let mut skipped = 0u32; + let mut events = Vec::with_capacity(nonces.len()); + for (nonce, value) in nonces.iter().zip(values) { + // `None`: the nonce is untracked in the v8 dust state, or was + // already destroyed there (both logged host-side). + let Some((night_value, owner)) = value else { + skipped = skipped.saturating_add(1); + continue; + }; + match LedgerApi::construct_cnight_generates_dust_event( + night_value, + &owner, + ctime, + UtxoActionType::Create as u8, + nonce.0, + ) { + Ok(event) => events.push(event), + Err(e) => { + log::error!(target: LOG_TARGET, "failed to construct replay event: {e:?}"); + skipped = skipped.saturating_add(1); + }, + } + } + + let (restored_so_far, _) = DustReapplyProgress::::get(); + let mut applied = events.len() as u32; + if !events.is_empty() && !apply_batch::(events) { + if restored_so_far == 0 { + // Nothing has been restored yet, so the likely reason is that the + // hardfork did not wipe dust after all: re-applying a surviving + // `Create` fails with `GenerationInfoAlreadyPresent`. This is the + // self-cancel that keeps the migration inert against a + // translation that carries dust across. (Keyed on "nothing + // restored" rather than "first + // batch" because a leading page can legitimately resolve to no + // events at all, and then never apply anything.) + log::warn!( + target: LOG_TARGET, + "replay batch failed with nothing restored yet; assuming dust state survived the hardfork and cancelling the replay" + ); + return Ok(cancel::()); + } + // A failed batch left the ledger state untouched (the ledger + // propagates the first event's error out of the whole system + // transaction, and `mut_ledger_state` only writes on success), so + // carrying on with the next page is safe. + Pallet::::deposit_event(Event::::DustReapplyBatchFailed { nonces }); + skipped = skipped.saturating_add(applied); + applied = 0; + } + + DustReapplyProgress::::mutate(|(total_applied, total_skipped)| { + *total_applied = total_applied.saturating_add(applied); + *total_skipped = total_skipped.saturating_add(skipped); + }); + + Ok(Some(last)) + } + + #[cfg(feature = "try-runtime")] + fn pre_upgrade() -> Result, sp_runtime::TryRuntimeError> { + // Count only: `UtxoOwners` is chain-scale, never snapshot it. + Ok((UtxoOwners::::iter_keys().count() as u64).encode()) + } + + #[cfg(feature = "try-runtime")] + fn post_upgrade(state: Vec) -> Result<(), sp_runtime::TryRuntimeError> { + use frame_support::ensure; + + let live: u64 = + Decode::decode(&mut state.as_slice()).expect("pre_upgrade count must decode"); + + ensure!( + Pallet::::on_chain_storage_version() == 2, + "storage version must be 2 after the dust replay" + ); + ensure!( + UtxoOwners::::iter_keys().count() as u64 == live, + "the dust replay must not change the live UtxoOwners set" + ); + ensure!( + PreForkStateKey::::get().is_none(), + "pre-fork ledger state key must be cleared after the dust replay" + ); + ensure!( + DustReapplyCtime::::get().is_none(), + "replay ctime must be cleared after the dust replay" + ); + ensure!( + DustReapplyProgress::::get() == (0, 0), + "replay progress must be cleared after the dust replay" + ); + + Ok(()) + } +} + +/// Applies one batch as a single `CNightGeneratesDustUpdate`, the same pair of +/// calls `process_tokens` makes. Returns false (having logged) on failure. +/// +/// `execute_system_transaction` deposits `pallet_midnight_system`'s own +/// `SystemTransactionApplied` event carrying the serialized transaction, which +/// is the indexer's hook — this pallet's variant is deliberately not emitted, +/// its `CmstHeader` being a Cardano position that has no meaning here. +fn apply_batch(events: Vec>) -> bool { + let tx = match LedgerApi::construct_cnight_generates_dust_system_tx(events) { + Ok(tx) => tx, + Err(e) => { + log::error!(target: LOG_TARGET, "failed to construct replay system tx: {e:?}"); + return false; + }, + }; + + match T::MidnightSystemTransactionExecutor::execute_system_transaction(tx) { + Ok(_) => true, + Err(e) => { + log::error!(target: LOG_TARGET, "replay batch failed to apply: {e:?}"); + false + }, + } +} + +/// Wind the replay up without restoring anything, and let the observer resume. +fn cancel() -> Option { + clear_transient::(); + Pallet::::deposit_event(Event::::DustReapplySkipped); + finish::() +} + +/// Wind the replay up after the last page, reporting the tallies. +fn complete() -> Option { + let (applied, skipped) = DustReapplyProgress::::get(); + clear_transient::(); + Pallet::::deposit_event(Event::::DustReapplyCompleted { applied, skipped }); + log::info!(target: LOG_TARGET, "dust generation replay complete: {applied} applied, {skipped} skipped"); + finish::() +} + +fn clear_transient() { + PreForkStateKey::::kill(); + DustReapplyCtime::::kill(); + DustReapplyProgress::::kill(); +} + +/// MBMs don't bump the pallet's `StorageVersion`; do it ourselves so +/// `process_tokens` starts accepting observations again. +fn finish() -> Option { + StorageVersion::new(2).put::>(); + None +} diff --git a/pallets/cnight-observation/tests/dust_reapply_tests.rs b/pallets/cnight-observation/tests/dust_reapply_tests.rs new file mode 100644 index 000000000..aafb45887 --- /dev/null +++ b/pallets/cnight-observation/tests/dust_reapply_tests.rs @@ -0,0 +1,338 @@ +// This file is part of midnight-node. +// Copyright (C) Midnight Foundation +// SPDX-License-Identifier: Apache-2.0 +// Licensed under the Apache License, Version 2.0 (the "License"); +// You may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! v1 -> v2 dust generation replay tests, against a **real** ledger. +//! +//! The mock wires the real `pallet-midnight`/`pallet-midnight-system` over a +//! parity-db arena, so the replay's system transactions genuinely apply to a +//! ledger-9 state, and the pre-fork values genuinely come out of a ledger-8 one +//! seeded in the same arena (v8 and v9 share the storage backend). + +use frame_support::{migrations::SteppedMigration, pallet_prelude::*, weights::WeightMeter}; +use midnight_node_ledger_helpers::{ + CNightGeneratesDustActionType, DustPublicKey, SystemTransaction, deserialize, + serialize_untagged, +}; +use midnight_node_res::networks::{MidnightNetwork, UndeployedNetwork}; +use midnight_primitives_cnight_observation::DustPublicKeyBytes; +use pallet_cnight_observation::{ + DustReapplyCtime, DustReapplyProgress, Event, Pallet, PreForkStateKey, UtxoOwners, + migrations::v2::{MAX_REAPPLY_BATCH, MigrateV1ToV2}, +}; +use pallet_cnight_observation_mock::mock::{ + self, CNightObservation, RuntimeEvent, System, Test, new_test_ext, +}; +use sp_core::H256; +use test_log::test; + +/// v8-side ledger types, reached through the helpers crate's per-generation +/// re-exports (the same crates the node's ledger-8 module is built from). +mod v8 { + pub use midnight_node_ledger_helpers::ledger_8::{ + base_crypto::{hash::HashOutput, time::Timestamp}, + ledger_storage::{db::ParityDb, storage::default_storage}, + midnight_serialize::tagged_serialize, + mn_ledger::{ + dust::{DustPublicKey, InitialNonce}, + structure::{ + CNightGeneratesDustActionType, CNightGeneratesDustEvent, LedgerState, + SystemTransaction, + }, + }, + transient_crypto::curve::Fr, + }; +} + +/// The fork block's time. +const FORK_TIME_SECS: u64 = 1_800_000_000; + +/// The `ctime` every replayed entry must carry: the fork block backdated by the +/// dust `time_to_cap`, so each restored entry is at its DUST cap on arrival. +/// Derived from the *active* (v9) parameters — the 8 -> 9 translation recasts +/// `parameters.dust` unchanged, so this is an independent check of the value the +/// migration reads out of the v8 state. +fn expected_ctime_secs() -> u64 { + FORK_TIME_SECS + - midnight_node_ledger_helpers::INITIAL_PARAMETERS.dust.time_to_cap().as_seconds() as u64 +} + +fn init_ledger_state() { + let path_buf = tempfile::tempdir().unwrap().keep(); + let state_key = midnight_node_ledger::latest::storage::init_storage_paritydb_separate( + &path_buf, + UndeployedNetwork.genesis_state(), + 1024 * 1024, + ); + + mock::Midnight::initialize_state(UndeployedNetwork.id(), &state_key); + mock::System::set_block_number(1); + mock::Timestamp::set_timestamp(FORK_TIME_SECS * 1000); + StorageVersion::new(1).put::(); +} + +fn nonce(byte: u8) -> H256 { + H256([byte; 32]) +} + +fn owner() -> v8::DustPublicKey { + v8::DustPublicKey(v8::Fr::from(7u64)) +} + +fn owner_bytes() -> DustPublicKeyBytes { + DustPublicKeyBytes(serialize_untagged(&owner()).unwrap().try_into().unwrap()) +} + +/// Build a ledger-8 state whose dust generating set holds `entries`, persist it +/// into the (shared) arena and return its root — exactly what +/// `RecordPreForkState` would have saved during the hardfork upgrade block. +fn seed_pre_fork_state(entries: &[(H256, u128)]) -> Vec { + let events = entries + .iter() + .map(|(nonce, value)| v8::CNightGeneratesDustEvent { + value: *value, + owner: owner(), + time: v8::Timestamp::from_secs(FORK_TIME_SECS - 1_000), + action: v8::CNightGeneratesDustActionType::Create, + nonce: v8::InitialNonce(v8::HashOutput(nonce.0)), + }) + .collect(); + + let (state, _) = v8::LedgerState::::new(UndeployedNetwork.id()) + .apply_system_tx( + &v8::SystemTransaction::CNightGeneratesDustUpdate { events }, + v8::Timestamp::from_secs(FORK_TIME_SECS - 1_000), + ) + .expect("seed v8 dust generation entries"); + + let mut sp = v8::default_storage::() + .arena + .alloc(midnight_node_ledger::ledger_8::api::ledger::Ledger::new(state)); + sp.persist(); + let mut root = Vec::new(); + v8::tagged_serialize(&sp.as_typed_key(), &mut root).expect("serialize v8 root"); + root +} + +/// Drive the replay to completion, returning the number of steps taken. +fn run_to_completion() -> u32 { + let mut cursor = None; + let mut steps = 0; + loop { + let mut meter = WeightMeter::new(); + cursor = MigrateV1ToV2::::step(cursor, &mut meter).expect("step must not fail"); + steps += 1; + if cursor.is_none() { + return steps; + } + } +} + +/// The `CNightGeneratesDustEvent`s of every system transaction applied so far. +fn applied_dust_events() -> Vec { + System::events() + .iter() + .filter_map(|record| match &record.event { + RuntimeEvent::MidnightSystem( + pallet_midnight_system::Event::SystemTransactionApplied(applied), + ) => Some(applied.serialized_system_transaction.clone()), + _ => None, + }) + .flat_map(|tx| { + let SystemTransaction::CNightGeneratesDustUpdate { events } = + deserialize(&tx[..]).expect("deserialize replay system tx") + else { + panic!("replay must apply a CNightGeneratesDustUpdate"); + }; + events + }) + .collect() +} + +fn cnight_events() -> Vec> { + System::events() + .iter() + .filter_map(|record| match &record.event { + RuntimeEvent::CNightObservation(e) => Some(e.clone()), + _ => None, + }) + .collect() +} + +/// The happy path: every live `UtxoOwners` nonce is restored with the night value +/// and dust owner the pre-fork ledger held for it, stamped with a `ctime` that +/// puts it at its DUST cap. A nonce the pre-fork state doesn't know is tallied as +/// skipped. +#[test] +fn replays_live_entries_from_pre_fork_state() { + new_test_ext().execute_with(|| { + init_ledger_state(); + + let entries = [(nonce(1), 100u128), (nonce(2), 250u128), (nonce(3), 7u128)]; + PreForkStateKey::::put(seed_pre_fork_state(&entries)); + for (nonce, _) in entries.iter() { + UtxoOwners::::insert(nonce, owner_bytes()); + } + // Live in the pallet but absent from the pre-fork ledger state. + UtxoOwners::::insert(nonce(9), owner_bytes()); + + assert_eq!(run_to_completion(), 2, "one batch, then the completing step"); + + assert_eq!(cnight_events(), vec![Event::DustReapplyCompleted { applied: 3, skipped: 1 }],); + assert_eq!(Pallet::::on_chain_storage_version(), 2); + assert!(PreForkStateKey::::get().is_none()); + assert!(DustReapplyCtime::::get().is_none()); + assert_eq!(DustReapplyProgress::::get(), (0, 0)); + assert_eq!( + UtxoOwners::::iter().count(), + 4, + "UtxoOwners is the live set, not a queue — it must survive the replay", + ); + + // The applied events must be field-for-field the wiped ones, bar `ctime`. + let expected_owner: DustPublicKey = + midnight_node_ledger_helpers::deserialize_untagged(&mut &owner_bytes().0[..]).unwrap(); + let mut applied: Vec<(u128, [u8; 32])> = applied_dust_events() + .iter() + .map(|event| { + assert_eq!(event.action, CNightGeneratesDustActionType::Create); + assert_eq!(event.owner, expected_owner, "restored owner must be the ledger's"); + assert_eq!( + event.time.to_secs(), + expected_ctime_secs(), + "replayed ctime must be the fork block backdated by time_to_cap", + ); + (event.value, event.nonce.0.0) + }) + .collect(); + applied.sort(); + let mut expected: Vec<(u128, [u8; 32])> = + entries.iter().map(|(nonce, value)| (*value, nonce.0)).collect(); + expected.sort(); + assert_eq!(applied, expected); + }); +} + +/// The inert-today path, for real: re-applying entries that are still present +/// fails with `GenerationInfoAlreadyPresent` on the first batch, which is how the +/// replay detects that the hardfork did not wipe dust after all. Driven by +/// replaying twice — the second run's ledger state already holds the entries. +#[test] +fn first_batch_failure_self_cancels() { + new_test_ext().execute_with(|| { + init_ledger_state(); + + let entries = [(nonce(1), 100u128), (nonce(2), 250u128)]; + let pre_fork_key = seed_pre_fork_state(&entries); + PreForkStateKey::::put(pre_fork_key.clone()); + for (nonce, _) in entries.iter() { + UtxoOwners::::insert(nonce, owner_bytes()); + } + run_to_completion(); + + // Now the current (v9) state holds them, as it would if the hardfork had + // carried dust across instead of wiping it. + frame_system::Pallet::::reset_events(); + StorageVersion::new(1).put::(); + PreForkStateKey::::put(pre_fork_key); + + assert_eq!(run_to_completion(), 1, "the failing first batch must end the replay"); + + assert_eq!(cnight_events(), vec![Event::DustReapplySkipped]); + assert_eq!(Pallet::::on_chain_storage_version(), 2); + assert!(PreForkStateKey::::get().is_none()); + assert!(applied_dust_events().is_empty(), "nothing must have been applied"); + }); +} + +/// More rows than one batch: the cursor hands off between steps and every row is +/// visited exactly once (the tallies sum to the row count). +#[test] +fn pages_across_steps_visiting_every_row_once() { + new_test_ext().execute_with(|| { + init_ledger_state(); + + // Only a couple of rows resolve against the pre-fork state; the rest are + // tallied as skipped. Keeps the seeded v8 state small while still + // spanning three pages of `UtxoOwners`. + let rows = MAX_REAPPLY_BATCH * 2 + 5; + let entries = [(nonce(1), 100u128), (nonce(2), 250u128)]; + PreForkStateKey::::put(seed_pre_fork_state(&entries)); + for i in 0..rows { + UtxoOwners::::insert(H256::from_low_u64_be(i as u64 + 1), owner_bytes()); + } + for (nonce, _) in entries.iter() { + UtxoOwners::::insert(nonce, owner_bytes()); + } + let total = UtxoOwners::::iter().count() as u32; + + assert_eq!(run_to_completion(), 4, "three pages plus the completing step"); + + let Some(Event::DustReapplyCompleted { applied, skipped }) = cnight_events().pop() else { + panic!("replay must complete, got {:?}", cnight_events()); + }; + assert_eq!(applied, 2); + assert_eq!(applied + skipped, total, "every row must be visited exactly once"); + assert_eq!(Pallet::::on_chain_storage_version(), 2); + }); +} + +/// A batch that fails *after* something has already been restored is a genuine +/// batch failure, not the "dust survived the hardfork" signal: report its nonces +/// and carry on to the next page. +#[test] +fn later_batch_failure_is_reported_and_the_replay_completes() { + new_test_ext().execute_with(|| { + init_ledger_state(); + + let pre_fork_key = seed_pre_fork_state(&[(nonce(1), 100u128)]); + PreForkStateKey::::put(pre_fork_key.clone()); + UtxoOwners::::insert(nonce(1), owner_bytes()); + run_to_completion(); + + // Replay the same nonce again — it is now present in the ledger, so its + // batch fails — but against progress that says an earlier page landed. + frame_system::Pallet::::reset_events(); + StorageVersion::new(1).put::(); + PreForkStateKey::::put(pre_fork_key); + DustReapplyProgress::::put((5, 0)); + + assert_eq!(run_to_completion(), 2, "the replay must carry on past a failed batch"); + + assert_eq!( + cnight_events(), + vec![ + Event::DustReapplyBatchFailed { nonces: vec![nonce(1)] }, + Event::DustReapplyCompleted { applied: 5, skipped: 1 }, + ], + ); + assert_eq!(Pallet::::on_chain_storage_version(), 2); + }); +} + +/// A `PreForkStateKey` that isn't a ledger-8 root (here: the current v9 root) +/// must abandon the replay rather than silently restore nothing. +#[test] +fn unreadable_pre_fork_key_cancels() { + new_test_ext().execute_with(|| { + init_ledger_state(); + + PreForkStateKey::::put(mock::Midnight::state_key()); + UtxoOwners::::insert(nonce(1), owner_bytes()); + + assert_eq!(run_to_completion(), 1); + + assert_eq!(cnight_events(), vec![Event::DustReapplySkipped]); + assert_eq!(Pallet::::on_chain_storage_version(), 2); + assert!(applied_dust_events().is_empty()); + }); +} diff --git a/pallets/cnight-observation/tests/migration_tests.rs b/pallets/cnight-observation/tests/migration_tests.rs index 4da2fb509..90fdfc235 100644 --- a/pallets/cnight-observation/tests/migration_tests.rs +++ b/pallets/cnight-observation/tests/migration_tests.rs @@ -11,24 +11,31 @@ // See the License for the specific language governing permissions and // limitations under the License. -//! v0 -> v1 storage migration tests. +//! Storage migration tests that need no ledger. //! //! Drives `SteppedMigration::step` directly on the mock runtime; the MBM //! framework is not exercised here. Uses `mock_with_capture` to avoid the -//! ledger dependency — the migration only touches pallet storage. +//! ledger dependency — so this covers the v0 -> v1 migration in full, and the +//! parts of the v1 -> v2 dust replay that stop short of a ledger read (the rest +//! lives in `dust_reapply_tests.rs`, against a real ledger). use frame_support::{ migrations::{SteppedMigration, SteppedMigrationError}, pallet_prelude::*, storage_alias, + traits::OnRuntimeUpgrade, weights::{RuntimeDbWeight, WeightMeter}, }; use midnight_primitives_cnight_observation::{CardanoRewardAddressBytes, DustPublicKeyBytes}; use pallet_cnight_observation::{ - Config, Mapping, MappingEntry, Pallet, + Config, DustReapplyCtime, DustReapplyProgress, Event, Mapping, MappingEntry, Pallet, + PreForkStateKey, migrations::v1::{MAX_ENTRIES_PER_ADDR, MigrateV0ToV1}, + migrations::v2::{MigrateV1ToV2, RecordPreForkState}, +}; +use pallet_cnight_observation_mock::mock_with_capture::{ + MockLedgerStateKey, RuntimeEvent, System, Test, new_test_ext, }; -use pallet_cnight_observation_mock::mock_with_capture::{Test, new_test_ext}; use sidechain_domain::UtxoId; /// Matches the legacy pre-migration `Mappings` storage. Kept in a sub-module @@ -167,6 +174,92 @@ fn returns_cursor_to_resume_when_meter_exhausts_mid_migration() { }); } +fn cnight_events() -> Vec> { + System::events() + .iter() + .filter_map(|record| match &record.event { + RuntimeEvent::CNightObservation(e) => Some(e.clone()), + _ => None, + }) + .collect() +} + +/// The upgrade block must capture the (still untranslated) ledger-8 state key, +/// which is the only place the wiped dust entries' values and owners survive. +#[test] +fn records_pre_fork_state_key_on_upgrade() { + new_test_ext().execute_with(|| { + MockLedgerStateKey::set(vec![0xAB; 64]); + StorageVersion::new(1).put::>(); + + RecordPreForkState::::on_runtime_upgrade(); + + assert_eq!(PreForkStateKey::::get(), Some(vec![0xAB; 64])); + assert_eq!(cnight_events(), vec![Event::DustReapplyStarted]); + }); +} + +/// Already-migrated chains (and fresh ledger-9 genesis) must not re-arm it. +#[test] +fn records_nothing_when_already_at_v2() { + new_test_ext().execute_with(|| { + MockLedgerStateKey::set(vec![0xAB; 64]); + StorageVersion::new(2).put::>(); + + RecordPreForkState::::on_runtime_upgrade(); + + assert!(PreForkStateKey::::get().is_none()); + assert!(cnight_events().is_empty()); + }); +} + +/// The replay is deliberately paced at one batch per block by charging half a +/// block per step (the benchmarked `process_tokens` weight says nothing about the +/// ledger cost of 200 dust `Create`s). Pin that: anything less than half a block +/// must defer to the next block rather than run a second batch in this one. +#[test] +fn replay_step_charges_half_a_block() { + new_test_ext().execute_with(|| { + let block_weights: frame_system::limits::BlockWeights = + ::BlockWeights::get(); + let half_block = block_weights.max_block.ref_time() / 2; + + let mut meter = WeightMeter::with_limit(Weight::from_parts(half_block - 1, u64::MAX)); + assert!( + matches!( + MigrateV1ToV2::::step(None, &mut meter), + Err(SteppedMigrationError::InsufficientWeight { .. }) + ), + "under half a block, the step must defer", + ); + + let mut meter = WeightMeter::with_limit(Weight::from_parts(half_block, u64::MAX)); + assert!(MigrateV1ToV2::::step(None, &mut meter).is_ok()); + assert!( + meter.remaining().ref_time() < half_block, + "the step must consume what it charged, so no second batch fits", + ); + }); +} + +/// Without a pre-fork state key there is nothing to replay (no v8 fork happened), +/// so the migration must wind itself up rather than stall the observer. +#[test] +fn replay_without_pre_fork_key_cancels() { + new_test_ext().execute_with(|| { + StorageVersion::new(1).put::>(); + DustReapplyProgress::::put((5, 5)); + + let mut meter = WeightMeter::new(); + assert!(MigrateV1ToV2::::step(None, &mut meter).unwrap().is_none()); + + assert_eq!(cnight_events(), vec![Event::DustReapplySkipped]); + assert_eq!(Pallet::::on_chain_storage_version(), 2); + assert!(DustReapplyCtime::::get().is_none()); + assert_eq!(DustReapplyProgress::::get(), (0, 0)); + }); +} + #[test] fn step_resumes_strictly_past_provided_cursor() { new_test_ext().execute_with(|| { diff --git a/pallets/cnight-observation/tests/tests.rs b/pallets/cnight-observation/tests/tests.rs index 329421e7d..e29d8309a 100644 --- a/pallets/cnight-observation/tests/tests.rs +++ b/pallets/cnight-observation/tests/tests.rs @@ -1541,18 +1541,20 @@ fn position_guard_works_with_utxos_present() { }); } -/// While the v0 -> v1 MBM is still draining, `process_tokens` must short-circuit: -/// reading only v1 mid-migration would silently corrupt registration state for any -/// reward address whose v0 row hasn't been moved yet (e.g. a deregistration would -/// no-op here and then re-appear as live once the migration completes). +/// While any MBM of this pallet's storage is still draining, `process_tokens` +/// must short-circuit: mid v0 -> v1 it would read only v1 and silently corrupt +/// registration state for any reward address whose v0 row hasn't been moved yet +/// (e.g. a deregistration would no-op here and then re-appear as live once the +/// migration completes); mid v1 -> v2 it would race the dust generation replay. /// -/// This test forces `on_chain_storage_version` back to 0 to simulate a block where -/// the MBM is mid-flight, then asserts that an inherent carrying real UTXOs: +/// This test forces `on_chain_storage_version` back to 0, then to 1, to simulate +/// blocks where either MBM is mid-flight, and asserts that an inherent carrying +/// real UTXOs: /// - leaves `NextCardanoPosition` unchanged, /// - writes nothing to `Mapping`, /// - emits no pallet events. -/// After flipping the version to 1 (migration complete), the same call processes -/// normally and updates state. +/// After flipping the version to 2 (both migrations complete), the same call +/// processes normally and updates state. #[test] fn process_tokens_skips_during_mbm_then_resumes() { new_test_ext().execute_with(|| { @@ -1608,10 +1610,31 @@ fn process_tokens_skips_during_mbm_then_resumes() { advance_block_and_reset_events(); - // Migration completes: storage version flips to 1; the next inherent - // processes the same UTXOs normally. + // Same at version 1, where the v1 -> v2 dust generation replay is the + // migration in flight. StorageVersion::new(1).put::(); + let inherent = create_inherent(utxos.clone(), test_position(10, 1)); + let call = CNightObservation::create_inherent(&inherent).unwrap(); + assert_ok!(RuntimeCall::CNightObservation(call).dispatch(RawOrigin::None.into())); + + assert_eq!( + NextCardanoPosition::::get(), + position_before, + "NextCardanoPosition must not advance during the v1 -> v2 MBM", + ); + assert_eq!( + Mapping::::iter_prefix_values(cardano_reward_address).count(), + 0, + "no Mapping rows must be written during the v1 -> v2 MBM", + ); + + advance_block_and_reset_events(); + + // Migrations complete: storage version flips to 2; the next inherent + // processes the same UTXOs normally. + StorageVersion::new(2).put::(); + let inherent = create_inherent(utxos, test_position(10, 1)); let call = CNightObservation::create_inherent(&inherent).unwrap(); assert_ok!(RuntimeCall::CNightObservation(call).dispatch(RawOrigin::None.into())); diff --git a/pallets/midnight/src/lib.rs b/pallets/midnight/src/lib.rs index 8a6c36225..e6b67aa18 100644 --- a/pallets/midnight/src/lib.rs +++ b/pallets/midnight/src/lib.rs @@ -25,7 +25,7 @@ mod runtime_api; pub use runtime_api::*; pub use midnight_primitives::{ - LedgerMutFn, LedgerStateProviderMut, TransactionType, TransactionTypeV2, + LedgerMutFn, LedgerStateProvider, LedgerStateProviderMut, TransactionType, TransactionTypeV2, }; pub use midnight_node_ledger::types::active_version::LedgerApiError; @@ -60,11 +60,13 @@ pub mod pallet { }; use sp_runtime::Weight; - impl super::LedgerStateProviderMut for Pallet { + impl super::LedgerStateProvider for Pallet { fn get_ledger_state_key() -> Vec { StateKey::::get() } + } + impl super::LedgerStateProviderMut for Pallet { #[allow(clippy::unwrap_in_result)] // generic error type E cannot be constructed here fn mut_ledger_state(f: F) -> Result where diff --git a/primitives/midnight/src/lib.rs b/primitives/midnight/src/lib.rs index c71d36c79..8bfc2cd3b 100644 --- a/primitives/midnight/src/lib.rs +++ b/primitives/midnight/src/lib.rs @@ -26,10 +26,15 @@ use scale_info::TypeInfo; use sp_runtime::DispatchError; pub type LedgerMutFn = fn(Vec) -> Result, E>; -/// Trait to allow pallets to mutate the Ledger state -pub trait LedgerStateProviderMut { + +/// Trait to allow pallets to read the current Ledger state key +pub trait LedgerStateProvider { /// Get the current ledger state key fn get_ledger_state_key() -> Vec; +} + +/// Trait to allow pallets to mutate the Ledger state +pub trait LedgerStateProviderMut { /// Mutate the ledger state - must return an updated ledger state key and may optionally return extra data fn mut_ledger_state(f: F) -> Result where diff --git a/runtime/src/lib.rs b/runtime/src/lib.rs index 64bebef5b..c6f7facc0 100644 --- a/runtime/src/lib.rs +++ b/runtime/src/lib.rs @@ -488,7 +488,11 @@ parameter_types! { impl pallet_migrations::Config for Runtime { type RuntimeEvent = RuntimeEvent; #[cfg(not(feature = "runtime-benchmarks"))] - type Migrations = (pallet_cnight_observation::migrations::v1::MigrateV0ToV1,); + // Append-only: `ActiveCursor.index` indexes this tuple. + type Migrations = ( + pallet_cnight_observation::migrations::v1::MigrateV0ToV1, + pallet_cnight_observation::migrations::v2::MigrateV1ToV2, + ); // Benchmarks need mocked migrations to guarantee that they succeed. #[cfg(feature = "runtime-benchmarks")] type Migrations = pallet_migrations::mock_helpers::MockedMigrations; @@ -899,6 +903,8 @@ parameter_types! { impl pallet_cnight_observation::Config for Runtime { type MidnightSystemTransactionExecutor = MidnightSystem; + type LedgerStateProvider = Midnight; + type LedgerBlockContextProvider = Midnight; type WeightInfo = weights::pallet_cnight_observation::WeightInfo; } @@ -1121,6 +1127,11 @@ pub type CheckedExtrinsic = generic::CheckedExtrinsic, + // MUST precede the pallet-midnight translation below: it captures the + // still-untranslated v8 state key, which the cNIGHT dust generation replay + // (`pallet_cnight_observation::migrations::v2::MigrateV1ToV2`) reads the + // wiped entries' values and owners from. + pallet_cnight_observation::migrations::v2::RecordPreForkState, // Ledger v8 -> v9 state translation (the ledger 8->9 hardfork). Runs once, // when a ledger-8 runtime (pallet-midnight storage version 1) upgrades to // this ledger-9 runtime (storage version 2). diff --git a/util/toolkit/tests/hardfork_e2e.rs b/util/toolkit/tests/hardfork_e2e.rs index da7d11a63..86b7f1138 100644 --- a/util/toolkit/tests/hardfork_e2e.rs +++ b/util/toolkit/tests/hardfork_e2e.rs @@ -26,6 +26,9 @@ use testcontainers::{ runners::AsyncRunner, }; +/// Genesis-funded dev wallet the test transacts from. +const SOURCE_SEED: &str = "0000000000000000000000000000000000000000000000000000000000000001"; + /// Generate a chain-spec JSON string by running `build-spec` in the fork-from node container. fn generate_chainspec(image: &str, tag: &str) -> String { let output = Command::new("docker") @@ -124,6 +127,20 @@ async fn find_code_applied_block(rpc: &RpcClient, head: u64, old_spec: u64) -> u lo } +/// A plain (non-map) storage value at `hash`, or `None` if unset. +async fn storage_at(rpc: &RpcClient, pallet: &[u8], item: &[u8], hash: &str) -> Option> { + let key = format!( + "0x{}{}", + hex::encode(sp_crypto_hashing::twox_128(pallet)), + hex::encode(sp_crypto_hashing::twox_128(item)), + ); + let value: Option = rpc + .request("state_getStorage", rpc_params![&key, hash]) + .await + .unwrap_or_else(|e| panic!("state_getStorage({key}) failed at {hash}: {e}")); + value.map(|v| hex::decode(v.trim_start_matches("0x")).expect("hex-encoded storage value")) +} + /// Every way of reading the ledger state must answer at `height`. /// /// Both the `midnight_*` RPCs and a raw `state_call`: the fix lives in the ledger-9 @@ -195,7 +212,7 @@ async fn hardfork_single_tx() { "inmemory", "single-tx", "--source-seed", - "0000000000000000000000000000000000000000000000000000000000000001", + SOURCE_SEED, "--unshielded-amount", "10", "--destination-address", @@ -273,6 +290,61 @@ async fn hardfork_single_tx() { assert_ledger_state_readable(&rpc, applied, "code-applied block").await; assert_ledger_state_readable(&rpc, applied + 1, "post-migration").await; + // 5b. The cNIGHT dust generation replay (pallet-cnight-observation v1 -> v2) + // arms itself in the code-applying block and then runs as a multi-block + // migration. It must wind up: while it is in flight `process_tokens` + // ignores every Cardano observation, so a replay that never finishes + // silently strands the observer. Storage version 2 with the pre-fork key + // cleared is exactly "wound up", by either the restore or the + // self-cancel path. + // + // The `dev` preset carries no `UtxoOwners` rows, so this exercises the + // arming and wind-up, not the restore. Nor is the self-cancel visible + // here for the same reason (with nothing to replay there is no colliding + // `Create`); the pallet tests cover both against a real ledger state. + wait_for_finalized_block(&url, applied + 3, Duration::from_secs(60)).await; + let head_hash = block_hash_at(&rpc, applied + 3).await; + assert_eq!( + storage_at(&rpc, b"CNightObservation", b":__STORAGE_VERSION__", &head_hash).await, + Some(vec![2, 0]), + "cnight-observation must reach storage version 2 (dust replay wound up) by #{}", + applied + 3, + ); + assert_eq!( + storage_at(&rpc, b"CNightObservation", b"PreForkStateKey", &head_hash).await, + None, + "the pre-fork ledger state key must be cleared once the dust replay winds up", + ); + eprintln!("[hardfork_e2e] dust generation replay wound up by #{}", applied + 3); + + // 5c. The fork wipes dust state, and the `dev` preset has no `UtxoOwners` for + // the replay above to restore, so the genesis wallets cross the fork still + // holding NIGHT but generating no DUST — and with no DUST they cannot pay + // a fee. Re-register the source wallet's dust address to start generation + // again. The registration funds itself from the retroactive DUST its + // now-generationless NIGHT accrued, which is exactly the path a real + // holder takes after the wipe. + run_cli(&[ + "generate-txs", + "--fetch-cache", + "inmemory", + "register-dust-address", + "--wallet-seed", + SOURCE_SEED, + "-s", + &url, + "-d", + &url, + ]) + .await; + + // The sender only returns once the registration is finalized, but the NIGHT it + // re-registered starts generating from *that* block's time — at the tip there + // is still nothing accrued to spend. Give it a couple of blocks. + let registered_at = finalized_height(&rpc).await; + wait_for_finalized_block(&url, registered_at + 2, Duration::from_secs(60)).await; + eprintln!("[hardfork_e2e] dust address re-registered by #{registered_at}"); + // 6. Post-fork: run single-tx again to verify the node still works after the (future) upgrade run_cli(&[ "generate-txs", @@ -280,7 +352,7 @@ async fn hardfork_single_tx() { "inmemory", "single-tx", "--source-seed", - "0000000000000000000000000000000000000000000000000000000000000001", + SOURCE_SEED, "--unshielded-amount", "10", "--destination-address", From 5b6bdc1fc4af8d2e254c1edb0b7feb584285c8a3 Mon Sep 17 00:00:00 2001 From: Oscar Bailey <79094698+ozgb@users.noreply.github.com> Date: Wed, 12 Aug 2026 12:20:37 +0100 Subject: [PATCH 06/13] chore: add PR link to the cNIGHT dust re-apply change file The change file came across from the still-open #2012, which left the `PR:` field as a placeholder. Assisted-by: Claude:claude-opus-5 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> --- changes/runtime/added/cnight-dust-generation-reapply.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/changes/runtime/added/cnight-dust-generation-reapply.md b/changes/runtime/added/cnight-dust-generation-reapply.md index b75a46959..0b16581ea 100644 --- a/changes/runtime/added/cnight-dust-generation-reapply.md +++ b/changes/runtime/added/cnight-dust-generation-reapply.md @@ -58,4 +58,5 @@ self-cancels rather than corrupting state: the first replayed event collides wit New events: `DustReapplyStarted`, `DustReapplyBatchFailed`, `DustReapplyCompleted`, `DustReapplySkipped`. -PR: +PR: https://github.com/midnightntwrk/midnight-node/pull/2019 +Upstream PR: https://github.com/midnightntwrk/midnight-node/pull/2012 From 2ff2b4406806933a16fe69dfc13e7c56a8acccfc Mon Sep 17 00:00:00 2001 From: Oscar Bailey <79094698+ozgb@users.noreply.github.com> Date: Wed, 12 Aug 2026 14:51:15 +0100 Subject: [PATCH 07/13] chore: bump node version to 2.1.0 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> --- Cargo.lock | 2 +- docs/openrpc.json | 2 +- metadata/static/midnight_metadata.scale | Bin 135037 -> 136548 bytes metadata/static/midnight_metadata_2.1.0.scale | Bin 0 -> 136548 bytes node/Cargo.toml | 2 +- runtime/src/lib.rs | 2 +- 6 files changed, 4 insertions(+), 4 deletions(-) create mode 100644 metadata/static/midnight_metadata_2.1.0.scale diff --git a/Cargo.lock b/Cargo.lock index 415d954d8..1c9424d93 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -7870,7 +7870,7 @@ dependencies = [ [[package]] name = "midnight-node" -version = "2.0.0" +version = "2.1.0" dependencies = [ "async-trait", "authority-selection-inherents", diff --git a/docs/openrpc.json b/docs/openrpc.json index 9b4e15f90..d758dabdb 100644 --- a/docs/openrpc.json +++ b/docs/openrpc.json @@ -2,7 +2,7 @@ "openrpc": "1.4.0", "info": { "title": "Midnight Node JSON-RPC API", - "version": "2.0.0", + "version": "2.1.0", "description": "JSON-RPC API for the Midnight privacy blockchain node. Custom methods provide access to the privacy ledger, governance parameters, and peer management. Standard Substrate methods are also listed." }, "methods": [ diff --git a/metadata/static/midnight_metadata.scale b/metadata/static/midnight_metadata.scale index 7c4ab497c2a791a1c44dc29d8c2c56b0198499db..49e1f4cdf87b62825cf07691fa85cd4c116097fc 100644 GIT binary patch delta 1567 zcmah}&x;&I80`wH&S3)zLUP!^7l^nEvqOxRWD-P@C9WjKC0UIZ*{SXNX1bW3s-~)D z#`e-4bKjgrJbDQJIP8BQf)}q5y~rN(7f1wO)$}a0cyKN=Q{CTJ@4fH6@1L(bC*O6x z|Kqv1a`fl(Uz**WsSgiXR#jO)3ep8u*(;qF`vp@Wof=vk0E)&y;)>}jq;k>DEsA}kvB_EzBCv6HG~=+zM$ z@fywQx>%pSdo2?ieypqN^#5ybz9FKZ?G;09v@hbl1KBJJ2gD{g9dC{B5;P#YNrSf2 zAU!yVxHqSGRpw(>U6Q3^scKT-;%ps{tRrnmE;UGkZ1uA9;(>+6q0k;2XYBB0^r$Jq z+n#)>b0#}XbRjopg3~kO#ppbTm)4Xu{q@yTGMMj-w;z1?!S4R52bMhKs>+~31&H;| zxGiebz;2y&u_}+!YB8Uw=_f#^K3qPT>#8+hGENt-zdZHSsh4OjJ(SF?Q>`wckRiH^ zpuyBc<>ZCKlGFTm@Yln`cvRZa=;(*Pul)4$?;Fp~6;F;&o``1`#MR@YZ^Vsr&8Tj*J|9t0PGV_>0;7u83&bPn)?Y;Ked#|_lwOQVYCU#nt zX1P!*HKI74SfPhmc}S1W7K=61S}N&a!v3z7qFp_xhtf->Kn2>@QA!Oh(c3@`>9*=h ztIXNL)j$pFk;kLrbZhDnl~SXon+Hw=Dy_q}wx-?`K5*iq8gOq;OayAgyg4y(Q3Xl` znGt|qtTm$HVq>q~tc_c^N)MdC>$a9uTB(5$aAoj6b!b2hQVlhv)ZsxjtW+waMoJJ|QK**cg+MLpPu$GzO+Sgw2UNIXW+zZj z=}+Cv-fc&T?hmOcTWaV38U58?2M}HB;IP_W$4)KRs&loSopQ4oMJt7B;Zn2{Rhxl& zm;U9=1gn4732dd+{M;&Sq%)gw)VNY;0@y%(L4W&ZLe|w|0-!roxYmfuJHS$)?$I}Q z`PiNk{r`blW5Rl>QY&6wDPL+ZO5#A>r}g)5CQ^F!nt*FiO|F67L`^fTfqG2qFW*dn z+PX_Xa;F+yFK$PrRwZf#>YTp0FO}?|-C*PDRny zUZBcaf8=J)0C$t<`7Skls!=Y{&iK5(xjaq!N%S46(YX^Vw)-F73t>I*^Ca9T=11auqcYnoh09SG*LI??Ry|hS zE|jaG)X%DS%@wNQW)w!dQL%;P5^nB=(UqvN7v`~QE|shK>{J{zbK6m&o~uJ_i32^v z|BmTQ>y4922ftX4nqhN03ac$59bs)N*J6b z&4$!`5$Bvh57Ivg&{ILlwH;jz%{eioTG$ z5K!Q;Dy?VLL;C*vSI=xLoV@>jEEX(HER>q8owGIIQx;;_tcCSP?Mk^6g{_#I@e9sC z7PfHgZ#=B24{1HT8ij=hHM)aOfpD}^C>C2p8_^Bsrr4_BH@9~+qO*cbJtl4o?3hjr z*?HM5ywY;m9XQ_f*l4!}AL_f&u(C>FghLW-u4*$oUl6kVwkT)T4-^=*$a33-czarp zY@9r~h+pc+U_l!a{x*!H8k*`nj^yg(5A)$xt-(VvdnFrg02OZI$X<_I^?Icom4<2H zjjVcVgH|dnQ=sX|RvG%2FoF2&(_O%J?YW+?-yYt4*e~k0%${9cT7C2+Mv755Q1LT? zEZVTVKbu4f>y<(=;?)Sc!J9rVY(-7G_s0o~Pv~eD3XY1|?X;qJuU0{ZnR=G1vIt`; zpy_3*K=sUaw)D{K_9)JVpdLSIL`58zBP`Ss&sMe8*eO)7q@!$jE{?*y8-iULc~DZQ zdJ3h3rz&5-)b8b66D8Y5H(C6>(Q|su>?gBd_*@!}+2yiXt`-`5M-BC3U^{V&)e7zb z3h8^yswKT7#M2h?A=9p{5{RJ15YyUmV9u^To};K5x3Fc~Q2qnjf`oGH$?Zb*5|G!1 z^CTS&2`YPUncegRxY%%9n^XVAmV4ED;EI3VUG* zd!yI{-C#7|$#kX)&LsT7GCW=pn{>hk=?V)_jO^6GA!}i54wgJLvyrnZKZ}hgG!Bav z1II?JU5!fF;0`@kt2TMqz*(VDE>xRg9W2(W>qt)r*4=CXGv%UPfk37R2!x327!;0& z)i{)l6KVo#$Mo>9n#9wvn!=kyYC1h#ZW` zYyZ0BkRSsN7U)d1c6DBlf`+X?=%_Z8THYX@MKcQ0+dxrBjbw4Y_OvGk7q377iGar| zwab`c>_EnLt%~hR<7T90!zHts>jfMSfM9NT3#+wKejCr-H|iL9H14=C(>4@UU*DAF zYI}gRBiojtnFTahcXoK8hv*XcHE1PO7RChba;01YAF5R&l9^~|g&opSGnpuFfxe=A z`>lCQAr=)-QLdJET03h|qg*THFJ{#js53j0mdOeIw%cK+6%)W=ZWZ0+Nd91SxsQsK zLd-Ho4w|=80g}UBjux8PL0~tRBJf=&+!=|?pz#zyzKR9+7}hb6$oq;}Bu!I|Xx}c* z4UQc$5}}629DzeXJ&$&`p{8isZ%|rmHHzRK=%?_E>;QsBWJ@NrH6W_Nye7`7$=AIu z#M#btbX*_9ZX7#)G#`$6F{@0kX%UI0ilD)kb!uTOt9;04DpcYc;Yyk~dJe4}cVY>q zPY8xD+G$CX;TZAVqV0M!Dw67j`nH1=5(AJ)Z{a{QjW~R)>gdj><@vd!|S#Cu2TBDgG?tZ0K zWK^d0V)8Ku*{Zg+tiN=swmZF$s0SWV+nP4=$ar{)=-Mhv`eYs0c;t~sl%9s9m#bZ^ zMh$6ZSI^)B2#sf`j9XA@@Q6OkV1!p7L^r}@?|-~gjyI$20`^pcWEN}yoV>PcBsPOO zf?t%kc$57s`4!27SP#Squ3+!eOz}n(p`D~phKZCJQuP;wOGHhDU=ZB|eZfAhZEX=h zM$ceen^>YuhPg@N0<;)Vp%~3GFj!-PPLODjfiA@6S|G-ks!{3JjbT0dJBVR*htNFF z6F3gy zRNv&Gr1elKsucF7^nBMYQ@XMVrt1h#DaRn>q@rj8E15x%O@L-z* z5h}sB3Xw6SQ)M`Ib-P^LW_b~9#4O7}zi^fGERMH0ELRGbfSF52jj-_;p@Eo96>AYA z!C94simTfwDQ?$ltTmvq7$gWr?Bp?Jg-8zR2V_k|6$q&3w4WOuARowy4)ctlUaig? zZ=W?LXG@ZHE%Pg_$gaY{Ccp28ex3+$nky`2yYA=^iYM3T7i5x%t`p^#%~)$d(y|NZ zYWqsds`F>C#aXCeff(W#_(JmrG1=wx8XNknF-cW%&M?{eE*_Z<7WH(ns;6N3AI8SY zl}lrK2m;DdN$H8?{$JCJ$*-pu$HP_F)z^CWG?A?9I??HoHOQbKAA#D?@78ufS@=$# zOnx>!Yt_I@B^+;ARBrD1x~8e=+ZmMc&{bcjz0$;SY@7}Ms7I&QGVHq*{oY@_Q)%pW z!9bsGwNe}DnNFD7u#$lfh&~@Yt}g`ZIOnREAI;!PBI)i!()$+hwDcb%^Hq?C10Y>XL?S42Is%wwbo{(T$FRtbW}R=;Dg5>I(gETUg!qd zPMDiRER=C*#z19_+9z04deW3w${wsLmG2M6sGA{zqy(YPLXvnuoZg>i4=L#?9+iJ{S5DY&$jb=IT0WA`8-@HBOT`SMDN@g5&Uc`qpCuvKwippp zLpx#?9b)X;cKqC!F~2MGtbx|Z2RX+Day!n$q#2J%1-QhjmTkdKe@VP?I!KxK0BQF;SbKFF9@F7UIe zx0*5U;E(I&bu5k2sl8^jPFk5u(+`{g;T(EVzrM5C*F#BT>`Np;RwG#Kh-ZV7p{iwk zBvsT~jd~5J7)qXsub zCJ9I{?n|7e3fV}{Rv>$U1wcDQTCYZ}0NLiM)zkn03_j#`y}mOh9sQC4VV>Fv7opL@ zDQ-NZcM!G8)g9gb6(np(n-IXyiklmqyjmD@8$fi7BMSgjV6}+VIXE*yV5)_ep>qlg zo21Y!H?#NYGft?XLx3FPAZm~x>4pKc#}!if1Lr&N?IT)OEbUua_dmX+-hPj^s+^th9JAW=L& z>7Jm=i1qt*dII`8H$iEgE|;Fy$#0Zqo7?+!`={W&V`p&$f*QXOl`_}r;;y+4{Wx+VrEM%et-x)Juc&ZL&InG7>jfG#kf=bLMxu>}Cob9O^^q8Aqma zr5f$4UKqzxOY5iDU3h(C_RR~qwKHdyb8F{LEicXG-n{TcZh2w#qEatI+5VjRXm^{F z?K}m|8A(-rJktfh_$GJ?GzH95I+zSC`m`QlNT|n8cK6jyj_}Q=^*8(mf;03uw6V~R zoqOQK^vdk{+{*0I>f9p!%FWKtpIumA&z*W=V__W%KlSPB-I^KMn!|o;zS0e(Cxig* zb9$Kvv~~vQd?L3rZ_xQUEKE2{!%_MR87Mo+X)sq5Z|R!%^6Zih2B*L}pqq|!P1qKy zF_bN>=58(ilAb4GA)Y8v=^V-+1ZP5>D0zG1QLqeiVWwf5_px{DQO^oO zPLe^J0T;50Ly;VTV4Y~B^{C@PpVKceuIE;l9$nnPI!>Vc(TzpggkS`3i!LfX2Biv` zBQxmTT7ReC`k9+OJ3qU6CO5mjzOa#7on2YLaym~_i1x681_mD_Z7>;Pg?Wt$FgT-; zwhdi`)WuN-6xvO_2g6VDuk(rk&4K&(ZUQ_f=r2Lk`_tDqGsB&$mUmZRsVT=w&tn)2Fa8!SwQ(3y-|8m$ZT&JE^YJ(8JJ&kA zo`9gW@99h#K5FWFuykkCtKHrEDd4^rnF?;4&n?cbFXq;l{&WG{^as}pMd$1U>PI+| zABQRZN9aFLKe^V1x>uzhoDlI5q7)cWpnj%jk_EhUQ)+4+;w4F$w%QS`7u(CY=71WX zC89aUXFfRkYCR^u`85O)e!Wg!6M?!*>rwv6FApj8vuhqm7mV1L=Pg@pM6f=WEQ9=o zE*bKDmdnktiH_1(RMRCGP)PvBsflD@O#?%blYgb2`o*<^wohkR3a{yBg&`U?c6I@7 zM)OV%5;lgl@fw7YdsrCJ`dWML33s3lX?1J=(y+G$nmCXlI1)E=5-tEPPqPgXj+#0j}q4K{@`K*Lag zPDdi5p!^c=Wbz9#E@bWbCsxo(Lz$II^g}#*$bCCFrI1Fb0w=@119p7s0!~B!T_tkFly9= ziO`#y3-E%0W)AAV0(nZ+@E5a+tkUFHL-6FuM-uio{OlY84UY|;@REf(&_P3f#ZmUt zXn-J)@A2yr#7<+5vq+$4=_Na3#s^;z6Y8cr@sS#wg>1`Hm|Oq=p;fQcAh)qflJFtN zA?v1HFYpJfWXrntEW&QFQ7e)6PUAzbZc&$JcVg$C{PfpPV`__(?hyI z(shF^x~y)*nS@R$S5kiILKdJS2Ujl;Idd4nlEu(Jg>|5Hyf^#Vi5P!H>ubgAVaq ztqO>M0qzE=8}9{uPfa5dGBfXtB~VDw{*Sw8+~5odOA!hgI$F0HajgLq$&aI2FQ2K? zXVY{1<_z4Yh<^`Vgf<=S062k?TRXNKzQ*#G&*+)kl(Dge9_n(o%s1!_2}``zc5K`x ztzyDP7B08z;yo$e#K4VlVK8O|TB{%*Y+ZnQSRA0SC+J*!39e1he*^2VBCwA#E$Kq@ zkj7>p$9;S1?&_Dc^-cEGDpcX;C9W^jF?-fvTO8AG&ATX5=q#J9coCq$Ne;p4)N}+- z$Piu)>?zCs@gR`-a}rX454eW$nZMvMPKG*os|w;Wh?Jn?kvV1P2{@X-5lvn`r1g0- z83_Dun-4u^K*$xE4{KdA)%X}BW}egH;0D9+6Fp}RUXBdNJ~Q~o2Ta@o_5vmbe=zU} z4mZb&@ZE3@(%40B&98HCzaPDf42Wa*CTPHDNtx9Iq!)DF4=hg_qZV68h)l894WHwT zwI;y>Wh%wAV^$BiXMiwmiN3<1Fa*?5%%6l>Oa+j?&j(s^VT1*BSavqqyV)^pjH!#-zJ40VmCp5QgE!*T>_F6J-$Mgme!aa}N-^C_=gLuGE9SjQ{2w+b-hC|&TEmNHE2G`-r zjyOOt3H&*x)7?yB+L1b;GwEUZS0C5o396!}bb5G}2}htF)cP6uEp-jcX!ElL>rg}M z%~tU;b$$E>NbwK$0w0-Dh^ig{yE=&6PPOqq=+YBpcB%~#3Xxw)bRX|V_i;h@!)^6| zzm8c!Bm*kW_8Qd%yWl%ENG%?dfHSV841~CP)VtU8!%%_XeYIx)k+&ChPyB^%$RsgJD+ga z|H+!%CBURvF(?+BjR zMYuV@uM1~1%i&Cu=ImnvS>^QNVkb;gA#(&!aR_DjZ|! zxD~fZhYyFA)am%ehnNBRW3fE^+yfqv`xbv z&o8GFyc1HL2{{o`0cXw(TqI0f9ljv?fkGH~_za?-I)!kS=?$`A&T$|X+zpHe!W8Xbzu( zEIsB9`FsYMja$S@MO-<;#Ile&mc%!l4;$FVh)ZYC=qtEf8MP7zO`~0SXH$?k^-;lD z4OFYij@G7sWC$?hWV3K-h}>J?n7Dz)R?#dbqd)R~Q`SK$IKOU2(T0uegscD=TF^PP zfne9owxH1BLgqYy8;)Xts+#}(&E6D2@bW&J!o*{c#uU*i*)=<2K~av!*S!XQ8^(Po zc&_In=os-RSa%L*pazJ~AZ?k8TJO?oH#Vyt6bsoC=!0?u$H8<97K&k^d_C8};i)I| zI;)1b6Y@;(ammb?g*w3-51W#_8?n)OE3-W1hz|MjZ57HD@}iV2eU8FQ1Q4)74y(xv zAU4>BgWrkz!A%g9MM7FZA)q5<@bl(#*0+n;Rc?-&Gb@hM;L1P*2ondvST9vNQaZ3e z$N0@Kq5hEG&<;^vh7Y}S)GZImeP%o6AXuiJG8h?H zd+#(rmhZR zXYaosR1!)fWbgoF?2BR~d^^!?u$VRMc}{lPZ7HE5>>}C5L#|{0+%|Rb!2RRe*E3tMR&+90>NP-K<@vn^s>f*UKBv+j9>9z_01 zR0cmHdYDz81|G!R0(wYApcD}FeL4uPenwJkUCe%N0V+^BFQdi+WeAcyyj+P)B#50r zFbBkbWBM7Q<~!r#&QJ(^C2^IY|Dj)Pqzg+{mN#z8`y zs}iA;6hvfx-wx@+tc9d6yNkLTe*__uf9GaKX!e?n{Zk|SX3fg2YzlN}0AKp;$)>IF%@M_AC2w1mW-2wES*ohS|1@jtPj#_Og#i z^I5Hbz?6XNv&uY45V~~eNvc3*%44(GLO{D*BZ{Hhp{Mpgz#kPP}KKaadomw)(;;^Qt$gAN=?URD&mWI>5WIB1f*6#vM zh*sbd0B!2Dd@~+WN*H%+0VyzPP_D7MdQxY406D?uVeClPPk9ckP;_-%A{Jv7ej6F}_MT`PPmFzkgdvH9#LJWG);kbsB2nV>WRM0d!N&C@6b%%$ zKN{OCIV}p6u@6eUqvy~PjgR@3gNcmp%gU0>D-8>{T@9Q}eo#JX{SR_vUBH4_zB(7JQh5hCZSD#QELN;QZMSkgoDHgN&N7^ zGZjs(wA*!qZaEj^9|4_GB8ozDanpFDY{)kKmEbR#a2A3!$(ulPnGtQ+!mb6R&) zpLQX=aqkMlr84rdH$RB>=J}UW9{}nPPpJ>~hFdr=wFONB&|NC_57SYM=ey5;w+-2mvAXMj4L;6g=?uFzy&WV zRIa7!AvMI29c}k~DOEDiE%+q#lHRjK_j7%WR&Btm_7iX<1KHi#p>&oeHE!I#CIg;w zJ&o8S%SFnY`mi36$B;3l9 zsT&O(mU)w%tjPr?&TaC&EnOP9^bAEjKo!T?m9~)O9zo~s_ZPLk1BX|apGR(A*`P#B zsQUPE(zTt1YhJyK9VMq&5M{F~o}npRq)sI@t|7l6xw zxMT2qFsX6e_QF9CCwT!CI#}2u;p{BlXAW3%o_8q}nF#ku4TG#+q=YazSiX&!>GQUsWwn64x*HO_xP7E==$FQMUn5CvEqFX4bo8BSTBL5M?Z7D2!H zIbptd-M*Ax)&XaQ`8LQT?Wzszr7!Dr9vi&pDxT?Fhrx_ng~q;*3)jGq#&(Lm!tC_0 z7|rcwN3u_|hQF!xvv&EqCa2vm(>TEUDhAmj>U~(Y*|Y%YzQd|kU)TEi#8#qI9v!u? zb(W(l8vpurW3W4S$2a%E-+b2)dm>P0M$CX=U~wq=DmIH7n0XJ9JVNR@gk`~-c4Vo_ zA`b3tRNp3&_+_oWyAL?M34uHhtFAnK+DU?S7JRr{oQnvs`2t;7$(XE;KeEKo{+k|OV+T|=Iva)a-)fQML)uX)%@ zq$DZGZJIrEQgtC*u#tcWn7CW(k9Vv{V)rh&Od>G|qNyo7>c>9ts(50j zFHd=jJO$t^0`Fi1msLD(p?fajCPQ&p{NtyH=~;L6?5d0b1#01#@M&TGtS0pBg`QeU zGv+#gton}4;cd)$SIUJjk^O3nlf>p-rYz$4xXtTmwgkN6UQY7yLnbRvH0UZF6IJ)~ z_Dm`^FGNQ^p4qjvvu7TkU0%G_}Js3fkP?1X8c||-j8BPA~qswLM|DbBOwymK%d4zj+d>63En=Y zJ0v^LBIE#W!%PBRxY~mFj+LDYNyo-!!ntG5Zeg-dG>qk zm=2JgV>*7@)~-^_4(D~J^(4LwNPu0yEdkjl2o^UO^EIxZf$tIA$O2}7O<@xW z(n=PDlJsS-I10~E!a{tr7FtI!9D+a!Xpv{tH#VU4_fFvyjtfHrJ1JRLZv@meSEONM z;pUs2ICyp*X$3iUG9tHk!VDV`Og`~ExTyIu4-CR)Pd0g~;vMA-$n0))FMRLmKdNDz z+B~e}VkV(wXa>8|yz$VpkF5!BQe&?R?M*kh(1Zq=7g_w6Uj(?pghDLLMF^5Alidz* z%)><)@ZIOkjKC`aB%k9aa^+Im?Ct2&?UyY$SK>wi5W;l++cA|&(eKa zBrpT~JSN^PH@T95T_2F1w1o;bwolMQtHh;nq%7kmPCDMJ7C*%DsazBxfb&R5tR=90 z16eiIc7epnH7v9-z8o?r|+on+OqZc*+uy3;7i;p^T1n$6HptEE~a1oBPJB6~D&1 zkbZ%YehN5W!T;AB-oJ(NQ=GIQZ{b1AA*D~iX6A$tAuy9iGS^-2g0l1BX_uk-b2!G^Z!Z_<(GJeG+Q-nwu30CLinxPv;D>Pz+$0I2{xQU#f zYGCvyyxc(R_sDS=$NfEFzLu5x1woT22H~|Y|`7vz)OvD5e`K5N4k)Rdh;s&mP__Y>P8b}iY zVoe)}6dzzgSn}jCjt2ytg@r)?muJoxYD-?tds!Q&GMTvtSMle?s zJ|yqz5YpkDLU=n%$s{ZxA78T>hU1g>==(iQU~rA_MJEBv-0g7C6qH~-w%!jO89UmM z`_VIFYe$8^#ogopDh zdXDgo8Q>+u;{uQ-z|S7U#z{!D544Az2up-SypK81l1MR7a89paleIZ>a&)7KDm2%0 zfVI^chIHe`f->-IHU+J6#sy=`4vn@M+)>Plax=pAFd`?a8{TUgFd#t^QjEMe49N6=^qM4ZLtP4hS(*aDMX!yB zf;i~SeY#cU#f+v<2Qxr|N#e{vBB0&-Z7&eXLT?PX1$fgIY&s|zuuE=aXf=({Qw{Q< zCJm<>VWBe4NKYPyK{6PR7bze{=vguqhL5u|li*Q@d5~aOOAUn#friP!n&%R?Y8Lbx z61z{tcv-?`7w!~x=|Z@~(ZL)Mpm(5qYnXuH_~CF8=OLoTbdtl= z(egn^FyaMS!t10U%?_D@1jqj$rHC&jB?1CHAp&6j6LJ-fsFFJDL-I+x7|B96E@N@59k4xr>>Vg3#KRBr|3ndodT7u*AJ&f2A_fw!Cjt9scRjVfIH-G9-XCGagu#^eM;nKI~}+~{h*c?pKT;pT6DtpUd z(70tVNL=21%V0nmjmf6>TWB!2JC%Voif=yNir{TTrf9y57Iv9tcVe{6)DAp55fNmZ zSP=TjR%yN4BRTd8SDi!a2+2{C=1WAl;+EM!kDF&QJtut^)r5J$9x4cc>{IU`pRSa; zF(#FJQkfU}V<;zqlI+Hlw6oyO55#4(+Z)q**+lX<3?u#Ny(xVt8M```4Bi+IS8M>z z3j0|j$v56p^acWgd%BtszEr^Bvqw`teD*&0X9Vfs6?K%nI`DKXl`+vMIq_zt&{~8h!bQFN>a=u4 z?rj4W8#{3hrgf}E7`NGL5Q+9XX-lTns3h_RsF9}=fqNU&1}kn6*Js+usxS3Ua7j~a zgZkilu)YU3#vQR0!yV^!;cVj){O+pF_$P3jW$zkzimxxu9y{^i8##Q=de1TB+^dHi z;Veb|`c-@kkn{NV+}UO0Cre%T{E8xW;Ayf&kx+{UF(r?_w64Ijmaa+IDf2}xp!E2p zRHkeBVskaAE>Ru0Gr)rZZc4psj9f?{ZT^_^68_b3b)3A6_koe{u&`?asJxeOazt>F zvLm|Jw$^hrecJMM6D{4C4Rbt|PUHcZT0uZt*(Ddj#Rek_r4tW6c>JM2mGojHzZ2<4 zD0X?h;cLY5J6S`kp@m{8F7(N3lTw+5<~F}PZ<5};IVu(lmpv2%NvjgOixjgr_s8K^!8468VWLMGE zTJ=v+w$4SSsfYVFwJ1&Dj@n`l7pxF;Gu#f{!1TBjm{w-{w=yHGAZ@Q{V~X3@K%yFM z(zLM9zlBL@!DMlq;Ljg586yQBOZ`8dmXC2Y4)Wte+LGQ^>?8LcsFjq?EHnxVKYHbn z$v|e~_gV+x(vCDnYem{Q9w3Pk#k$Mm34To3#|QXvz&=j$V_+X2SuUeV!Ef#@w^Ocho%#eD}L&f$t6OS>PLY&jQ~u_bl*b-Lt^=$L?9+n{dwp-*Nl= zpul&+JqvsfxMzWH(me}&54vZ8?~U$R;QJHzEbvXaXMyh__bl+8bk73co9y!&1-@zb zEbu+-o&~-~+_S*1-?1=Ebz^{XMu0QKL3fpciKG*e2=XE{?pfe_vwIf!mff?!x8j}!zE$@u@SU;GQv% zzuP?vd^z_l@a5gJz*lh30^g>47Wj&4)Y4}Gu;dE_!pIj0h+DouV7%lD1jub)AW)Wl zfq?lpeSyGv*%t_)6<;8b?zn>E0%_G32&6S%AduF5fk67CFAzu@zCa+2eStvQ^aTQG z%NGcwSA2m$des#ern2h`1kycUAdtSz7YL+(k1r5NpYjC)>DzsQK>GLk0)h1J^92Iw z)4o6;eTOSB?B(zG1p?_m;0pxOXMBM``c7XUkp6?dKp_2xe1SmvE?*#!zS|cFr2nul z5J>+KS6~>;v%WweeUC2?NdHk^AdvoJzCa*-&KC%z@AU-&=|AoZ1k!)P7YL-!`vQUV zeXhW;oPW|62&Dg%FAzvy@C5?ti@rc0{il6_K>E-40)h1XzCa-TfG-e8|5;xkkp6S7 zz%ZYee1SmvL0=${{`0;-ApIA7fk65pUm%cv*cS+-|DrDtNdF~YAdtT73k1@SxB|n5 z{$*bvkp3&aKp_37FAzvS<_iSUf7KTVr2m>P5J*4n3k1?n_yU3SU-tz9>A&F$3`6>) zFAzvSA&L(1kzW0fk66MS72Dvzv~MG z(tpnv2&A9$1p?{keStvw@B0FQ^gr+g0_higfk662Um%eFhrU1{{f}(Hi3y?nU-AV4 z>6d+hK>8p10)g~D@dX0uSA2m$`c+>bkp8E>Kp_3ke1SmvHD4f*e%%!q9`&F50)h0u z@C5?tH++FW`b}RTkp7pxKp_3Ee1SmvEngsze%luar2n-q5J>+US73P5cYJ|B`dwcj zkp8#6Kp_3^e1SmvJzpS@{+TZjNdJ~E5J>-fUm%da>I($Y@4EuSqy7h9Advo#zCa-T zfiDn9f9MMY(*Ma92&DhBFAzw7ASANvA<^ndXM0_p$i3Jj0>i7yaHf9eYa(*MmD z2&DhJFAzw7<_iSUpZfxV^#AY$0_p$h3k1?%_yU3S&s~AxQU8}O5J>-TUm%da<_iSU zU-|-p^#Ab%0_p#&)MNd1ALF9?fG(&Qjj`GU8lYlibRKYyBQpmVDRr)YGZWH`L_3dj zJ8y}~m$sWmb@N0jv*FSoR*@Yw*Px3AccOXukknYYt#Spr&T0OdG+C$lnUq-*CJZ5p_78-IvA zQt2T^ik`BCl$KNvB46+j<8n$%1*J!!Po~)K25u?|&qiBhPC@2iiuk58Fw6a#-$1 zd|H!XY}|$27b;RRio{Ilsn*!zXSGA-fG1Px$xdAGl~%wPawDXJ1>H)a{b{s6REL&5 zS{TqnYxD!gH@@hqmD0fi@+CKGxLBEz>bk_8tyH@A(wMNhed(es#MBLoavyrrl&&W= zqbE-?Fw>?!nZrgNMFN>p9U9mVru2p}8(xw!Q(X<<5LPZ#d6lY5Jy3c;KV?3(5%FCg{DE3yZ8;~^ zTqy&eY~Vo*@PK~CVb2zKeLkpgqZ*mG4MGBSWI*ql@671CKDtwlu4Aj0TDVj@Pt=DV``x88_=(rj(mh*#~0&>sAZRcWQ`?c+@$p9#~)E+SOmrb zEg9V48x^RdzGd$+)vx7bjYP{w)uJ<4x^)|(vj4WbfI-FU+eM!jZ zT`sP-ipZLe6s41vOW+o*t=ij>%+j(7{%#3{um(RjTTOW96*)BNCf; z1$Sz6SC+)iq3@~<0?90dC1tK^ZtQX7K7#t|b)@_$H?u*VSETdUkR4;lV)pGFLP@V> z42efXy+oENWUzun^p3>JWDyCQ1-Em=IV2K#C-So4Zj2cBee@Nv+_E+w;{uXe1L=z) zevF^7DTVN;A(305ic(y^vu}&irA}v_h|sf2uOjIMZsTaPSG!&pWHU|6y$Gm20S^hJ zVUXMy)a$8&No|2%hgWLELId@l6jCMfZ)sqFc!&t7v}lpws(J~zp2t^f$S@)TDiU|h zY}9HP#Gb7htHWw&1O8*xK)pAm$M~l-c$*rVh0oeHTt&*oK)o-er~R{3Nvp|3BDz8) zP%ozRVq1|^JfcPycDHfSDR!m0UEyk#Mx#|Hp+o(i8b4PpTmf02oT@N#PEk4)T*)_E zp-Zq!>JBxyf!mw|^?{TgrC-wI?<=4n=>X7%4Z2vTjQzT6u5_36jhio?eQRs#P2tMy z`P|Bx`E$z)xrMW5&z#LItj?X8UzpFWFa7DlMFmtOTTWs{kIkHhJ4l>_8B=x!mjhco zW6CB@MF4(<(qOlCFl%N?=a?=*V4XNuy^^0%U8-|`#-fu&I7c_Uu?rfuesIfiDgQ+dp0VNB!?zPR%^@1eh3bMTTfD% zgpl(%@~TcPWkj`1)!b9CA#YYiA0+boys=(7^2p2XHF}cZTgpQ zj`j%0^Ekv{T(Ud{)VOJJkum@Vc-^X3;LX>Js5;I9#PAB$sMd<S06YcB0W%wRJ4tN%y zSPI_c+cPW6+<<% z!BZRXxRSe+^T=d}JZqfE4WYj><`2_Z^&=^TKk#KcKZZk{QybW0S_eMD2~af5XR*ny zGwKtdj`V9jYJ#$W2!q3ue;HR18G$77Qg+IIHy_n2HHemCQu83wvH+E;i>wv;QeI})6+@ozE6&%&GlE5B7VvBX*J{;qlFUcG2QVTA6Rrs4C zZy)6*wC;0A2~UTaquCh?HF>jQgrMG^1%0+pw$p*&E4E??Y6BewCEI~u!ihSO8YZf8 zBI~$HGK!HY;t4QJ65W!lH39Bf-Z>jxMWRdy zz+X)11is1IJ5pXj>~+gDp!kKmL_5?xx)FGnRL-`m)$M-cs) z8tEBKDKEtwGAdzT#7QU0PxbAj$HD}o`xUzSOy91$S1_(6+P5dHz<#^5YhnIsP=CH} zM;!|l-IyC+?$mm)b^OGX(vZlD5wh7ek|_UT-xhl=;se1Sxf()IQkfZ zJ6^T_)}G1NyU62Eu{pmbw&$-3^XToMlgBQTPSw`zhU4 z0vt}wL&PC(Wn1TmDcw;<0`3Zqb~s_omO!!t^<%VSN)|xaNF19o&yv@xSDgXX7u$jQ z^ZjNzP>b_8OF-hJ^Cd(^x&=cWO8Qv5al^8ckmeJ@WQb_3?D+&Vr%37(9SZJ{ zVH*Oe$tU=JE2007sKe zSSD`ij-EH)6E`VWTU3v->(U^r(Jt~Y5iHrY1a!~=vOEo~1Enq}fyL}mJrZ9oLrGzu zZyUN(6FN;7k};Y3&71E?bR#j|0A~Sv6K&>)UV1SZ$ArkVF%zl0s-Oz+fMx9GN)jP|jt_)rl014gAB495<@OkmZs3MADqggrKx- zYLSr-lJIMsF!FCb{k*w&?bzzlql+8K`M1MBrm8j+BmXI`hi_YHGBB0Ky*>%tz~f&& zY*kM(^4$`&Z?2Z>X5I0k8AtWk+;Vd3h(#i>CrVQ-Sm9`EiU$M+T2;au$!k3K^LUCO zM&b=_;<$vFIgt8&{~Cxz2RTUV^T^;;LY7IyFpy|x1(sjv(ww{ykq?A`+0BGVkF>!0 zbLG-c5$D4k8;9(lrn>Jub1%J2n7l>JXhePvHHz?ky4O%)fqB0DlinKFBQnhGjipW>1XseUj$I5gms_&V0tJ>+o8D31-J4RF1pt1ieJ z{2Gx%A3@~bgB~Jb1e0*9vnFnZBP+6b&Vq^(Z7*<~Rf}@Y;!q04faJ)5KPRS zuhvq(SOOP@u8jz-fdgRTeuuolN}?Z125m}30?6ZJE1G&&3%Z*Z2wCO=iLfF56Ru+n zPdGFsd}Z{|%}>N;8IzFpi0hGPkU+BgsLiNDN&l}#l}dK94Jeh$T<9FGaNex?f%J&` zLZSw)R49l7f^%fkf<(Hy#C1%-2mCw;iI|J*OecZ^LvUn?K>tC*=B;P>whO|+I|}bbcl3w4s=L*b^}biLfK8f4TCwH zSxyed?yD87BqAWmtxSs?InI$grDd#nQk!*tKG$K}E+Z;rHex9|y@+EP!Uv0vLEv)Z zU4)Of8pwXVg1f`b^@ViyXiYuxNbrO8?OF@B6dWqSwrVjtn?Ow>cVj^PfR71Y5@rJ!y~7Oi8Bn zgn6?>)*eV14{6d(no10jv%QAqIKd1F3ExdfG2xWv!hXXIlZeh2IliB_rR8lHAXO!u z2n~E2*YdO=e1n*bqT9z>Q40rAoyG-CsFn9{~(e_2HcQ^2_IRH&}k$LFr z(&f^f1A4|@CW_j0IWS(}c7PaR%XbYx#bTWN&THknK-G_CfH^Xgh&Qw3m{(2bOKZD_ef!r&AQns z_;dp8<)US4zcPG>gNK>o$TcT4T85`3<{c&LdphoUB>9M-gYzP6$VC2dp<5+n zvWUN!C}qca7H=4Yx1}#$1BA5Tt`$Pk8etTT==~(<5$dBV->rjs(w0b(%KJq!E@vxd(+l%V&cLcD4bl0-MU-Igq^9CEazeOPcR>xeK*~ z^x)KPg*h%x?Of9;1CI_GEA z$X=xNnmK+@1)_xJ0+`^_EF=kQV67m;5ab^y!;zYW5E?VwCLp`eSrH~j-stjlOqCaM!0Y;?XaLHcX3J;R^rO4LUB?d@-42uz{d zd85)`Uk`3Qg%!)!3W{ESs_T;FzPAb{ZVsPnzc^gs2o5b7Crk{-L^2~W{U~u1@!gVM zL0re6%#|9(0~{CVk{PuCcHrT+}DK!}87FB_@j=_hshScnH?PDtVoSOw!$C^(FIAV2HmE|#=9mlcP60v<+ z9tghH)-uWe_OpFjN4hf+#?$-EP5)DQ5ZvqWPXn^D{hR2lfs;H|4o*@K`txTl^DT~$ z_Hkn73pROGk;fbakXj=l{bAOHd<93=KjDjlw{jXZ|)o8=j+EnxWz7 zzw@qby&ewE0X)0E`AjPJ40-^$97NZf3m(cYvdwjQi-To@9L35Fz&Vft%I${BhT;Io{ml3-l%^ zEJeG}Oo;J>Bj7PRDQn*N1mP?Mdy_plCt5<#49PJ2w`KNEOjj|&Z3P;$Z*5$y+1rD} zVDmnNVD9CnzMRf(7Tw!;JsDi+q*mgDRg+#DpSXh{XZGvw}L z4g;p?*OTuMte@$z z(%y`8Vq7^SdwB>o4lD6=n(S= z|07tt``loV3Z8aDj-;;Xk`9NP5e`YFrL%lx0*t|ZTEc<86Q2XY%ZP)9f8UidA_nO| zY>%=s2sjkum?C#Elw3C~#`Xm`2ucoMr^J#Sv;D7u(c^$!o|HL6I;doEaEwu!_o!G`I<%F(h(cl&0HxZxn^ z;2GDMzk;A0oCA^k@BpG|UO_g819Rvnv0K`p%=fd^5**k}-g}ek0LI34VKaD%&Kh)T zpmKXafLEQ#oDXfFX+cx2{RORGI0fYs4qD>VGmfDlkI1;3?lm z&z$i(s3&scdo`Yc{J^PP7-&xdWz!mEra)dRdlcx)$u$Bc%39bwCr-o96ahYLb%Nb< zYN?8w5PV5WsU=z<7qwdHX#M;$91M+>(>PyBHjcEOAg!8BkdDM$xWl9r73mI=ojNo& z)2#=P_e76!EwYzBHlPvNe|7rx6emzFjzihdzz%zCAjqH?MmBwJ0P)TE3z@y}*P)bp zq5~RuQc|{!w!{-WC^FH(UHZBbP^gOo9l1&>6#gAgu;Ykp`||SCIxGGEZtx zK8E83r8t-}yYq2*R4P1h&*@Qe(qkuv-kVa-8?g!*4s17|+Yg*j?YrJk7(K!5b<`Xz z)XVYP2lU7+{som+dO4v=>uh(*1W%Yo#92$d-4lm*5OKg?$bdq$>YW1$e-Kvx?tv-w zZu}a6{|B;eyr*-Ch)cr|rzBuY{-`yuu7iG&*9OonA!086CX5X*=_2^OwKAI;o}WNzf#;Q<)TYd10;WlL=?z}*9}Bd{GT>z#2!GLJJw7p^CPK1ozNCX6&{Eg1bNZZS`+V_&Jn63 zdI5GOI0=!gO92d^(~Zc^aGo^a;0$mSqgU-+=S{3!b{Xjl=cBCxY!AdhYB3f(beONL zx8S=_65m}Y)5$T2A+t%2NZv$j;jxl@3)$Oz9K!9ikex=elmf)%IShga3FYwrhYt>X zRPia@IRfGY_XA3affmT^f+S=L2mr`5UfUhJC$R_~)>{pVR%7QK$qY%cn*m3V;6!o> z9w3rH(zeA7@o_*L*-FM!5{*=PrLj9xy;MX0N{kOXbS=y=_G$C3$~1_YRAZUQKy6MQ zCh-OFWtlkvoFgIOr;(jst-&md;8!_AAplsx+OS%4e%yu%;6)ahUi_0v2hRhEV*F*t z4(@}!ny_G9bJ?K6`_8!ch075X;J6P78(O_@px5Ol zLyU(Zru*q(3Bb}XY9!||d`rJL03tdpRQ^R|ceFuTbZd>Uw)lf!QY3x4kHC*#?AhHi zZdKckDD2ZSK=>$_GWY%9h;g3PYLHyIbU5m6(9y?JLsYS#KR|?R7(>Gk6^uWadMNd9 zYBsfyT1q{ZI+uDPbum>)sSo013WNCY!1dmy^-{2ocGX7)d`ioYP-8Etj}4?3ky8h< zIC6l+tvDA~YR!0X9e>jomcM*rAU$Oq!A#PY@pX7`MYc*68xR^!S%e)8>%LmD$+HIHO$X zHdq)8>ez`pb@sBSz%zt2JbHF^b$$)zLpD)*+G$5-Ogz=sHxZ6>CHl}MYyz$h6 zf$WUUN&^!qG9v}SaXm|Bu|&EW*l=Mt!{3y|tVE6o7~wWZ4o~_%fNsM-Gvc}dr2Q)*=sF?$B&Cv zOQ@M-OkgCz)5JOQ%oloQP0D_LAUz!eN34kv8Z}8YDuz72P_H2u6^NBR+p8GtdhIpMI2{}|j{4+Ud(j;POjPW8XHp+EMUx{3ZKHWWMeWN%+?5kz$ zD_(xsW{zYWR#0BMVgr>WGq|+@<+(6Q*Brm|fo8*>Qt3FaHaq^tsrOI1pa5o0$_)83u0A!SJfT?A-d zMm@zvIPm%eN%%x^Dxs8;mDbJHN7tnB?HJzgQ&P(-$j%859|ykr10+3-D7y;DRB%16 zAm(O*a&eX-lZo?AHSM>Mtd&1>Sw!Xej}&I?dRBX%$<;#_RuQYi)7AmK0`=m{oGCS8 z$2+Wbv*!g+%5X`fxD#w#j%n&);b7jb2jT;#SZ8LII6>J)vtrDJAbb~G8Ss0$&{(RT zZowrJdL*?hugq8$b`b*|18QvM>w&9l-@i%>C9&`kIte(I6Dt&uEF%D!VMx?<#VNIs zD8?eQ8)Ts8u0?)RvrCP5tDZWAxLAl7e!`H;n34jUdLYu2NLErEQxY?zJ1w0iMHJTi zTX3F%rz-E0I+Q6yuNQe^BSMC<=b01C5hSRk6pp+@fO5|qw{-GF3A3SXfmOf=-LtL=_j0MSkU>s8*ojV=+kT?`E5R8rE$+ zbyQJqNSh0q@rM5S%llQuf8jZ?efm02w6P3;hQVP!Ji!0Az%2u1gB;Ni~C&+Fte`i3aAhFZI+ z)C>%-gB7$h>7QO7oc$<!(XX)yZ%Gsz;A@&y|$_y@RZJR);r)Mh_ z&#ima(dq?i!sk9_lG>|gO1;FKC@OVUK36T%mOxr9yZr`mXf@21VC$|Bs9nc$a6zq~GF_S@S!)3lAt#hVbo1isPM}h-0C_6Rj z)Jt9J1u(Q(v4=xadX$n+ROuiLv|fkq6l|hRQ5ryB)l+wh%NG(dMjzd2&_j26s8e@( z_#oXW^aYmL(TerD({z8`>5<;L)7v|Ar@v+`=^erKWQm3%&YUGk%aEB|FS3eXoappN zp}I4##U)ceH@Q;qd7Hp=kxUI1isJT1b^Eu9yur-3&+3=G-oCTR@vyB$*a7Yg(oexo zY7=zv@q=<$!C};1ls7}JqTsaU>N+cr07ynxT!%R%PC@u06sI=N>?bvU!Zr=7i`5$? zf1OsYUW2M<56;i8=x~jH!~Gf8*d-hheY0`z>gulRLVm(4CT(ReYhig%zjX}@F~U`U zVDT5dqhgUt;faBxb;y{vImI~a+xP%j&lsujhVb}>RSI5C!aX?Q zx8E6o88+te#4O%}q@v#U6ggr0Oy9kM-X-4yn6yZC1bvQ>huEx{d3dhC)C>ws>N(QK zB_{r%Kv(EXS37xLjPcnx;~YurK^I#9hbI*BBB-RgC+N<$n}ip;>0mcmb)XZhQujh9 zmbwqO3ijx*)OXOdo7cQG&*xn}8h}1qlc2YN=)uc~JUoWzS%(cATCo?8_UN`(17M2D z1oy{!eAd}C?gI?OtvuN$sK#*`GOoCNuty_s5;Aj3=YEV5nbdHJ(+FSUG_>>GfF;gyFN2Zkl_?mR9E$XO~vtrT^&a?8dpX3%TWm zRopoCSdYef;-Bu=_QrTSAPv&Lz@Xm2zup<#t<^hkK*R424$dK; zkggW%kTK0Y10?;4PR)8k0LIs5k=m2*CkWrdQdd}ziwHPel^hqw*|Ua1_$%=Ej3{Ry z`3>Rzydn~?z?Bo1@D(9tnF^g0vMUjNO2!Uf zxB+A%2nAXJ$Qr!W!uvz2pIcivo7*^_JM;L$*@g9urIp!@h5U6K9dWR`wxc6l2M#_u z;utLs9|^Y`I25$I-M|+k;T6)u?dJA=N5|jme{_V!s)u?LX5V(Vo8RBp!R8Non(zjD zJJ{T*4i?eh6Qrk&=P6vrki6qf@u)dCoCGAWic z*cnR#GWlMnJi(NDUmINoVCg~ZT5_Ttq>L!IxK!xsi``h5?2m%sjWQgQn|iPl1H%Zi z!ix_;@JNC$)%P3!#7<#vGs?;5AH;E^2NFyxT{u9X3w;aJ_Q1pk1QUbmgKd}qgq@h6 znn2x$`Ge^x^^qGuvOgB|zZS!(oWfy{nN#IUAjvPmQq4<{NL{m53t?!aRE6*ZC2wd@ zL%A9`J7MT0bW`Mw5`4&gNWlpO<8R90iB+oNza(pgIa!k7Iwh&eThQH6pQxPhrKgTT zXvs?lq@SoS`;KHO894snb3{v!Yo*;e6tzfAY*KnE4&e2PTM00Fw+E)+0ZUx(0Rnpj z{+_NaOe=qoXw1430~N5qhBZj4xCfrSgv~yTvlD>K?JVe9kLflDO})O5B9T|GX2JDv zE@sbNx&xR4Bvxojm?h>ZfOK(oeKB`#4Gz%r3%OJ0mfxJ4J9BPzBhTTfi09!nnx=cq zhy-Mq^!4z~gj!Yvctdf@kCX)yrIHAC;r6B=FU=V zA<&t&E1`$ho2}wy@;CWR+s^)M_s;&T?Cj6>>SG*zz&^|}13B-{8yO(yz_0;lGaO~; z%aL+U!qYBveF8t=sP5jw%@kQvaq*vP5=y;YsnbcZ)aQEvISL?!6VY*d#6;hGicSG- z7$Q$^9?54wfDHvrj1QEPcGp&UH;O0aRy8`~(d`CGeGy1XeHnhrO#g!StFKbQ*WnJ# z1ykRI#ya5O8Em=BFba><{=fF#1-P>NzVAB+doBc8U;|pq1e`3}L(Q_nlEC$nUfSRq zboK#|3k&-o7Zk~mYv3+$u?rC|;sV@-Dov>wDWPJk5o0P6Z7HQ1Dy3R#LV4sFdP+`3 zS4!on!d}+r9a53EF2;_k(FW^ zthm7ba8=Zw-;qU2@6n$gi{l^^pL$DpbTi6E!O*N;J|{C3Csj1mO)t=H(Glmacu1g#sx(Lul z+bIlx?|qF6zP>i7B>>JGLpe+7)#Nvwr6pK%jLzHlBtsPjk0`cUtst)DTFXJ&q?>G# zec$7TuT~#gptc;U9p95dZ`v`Hd?sY#6NSF!FM9&lVtMcRR+d~>>%=HttN}wPMLDuT zs20~aReb#cC-|fwElYVQ^SRW$&_53sPY3s5n|=BQ5nR z=(iOB7dvn>T79<=>^#)!wMyt5k_StaT1c8A(zO%!Dm@O|u%6p973H`BCh~?_A#I@! zwfMfyIH-YqvUSR8Ah6$45)HIxB9$HxSRb0ijmJMj?5C?OioynX&Z>oZ>i1-yuC}>w zo+xs>6eV;anpEVB(0_T71m#}Vf_P)?YBiFNrA~2gs!-gU1pkkv$<4L?{Q)r|f-P)y zZciRLBKV1I0fxz6P=KL_36Kt&OXV)Esf{#m;R8ym%nxF1f9O!g2LqRf|Hh-$ae63C zb6C92qd$sK2!AkM&>iIM_EiVlW$A_&%zHlEydS~R-Wdj+*dC=BN_%nJJ|pmq1-e2S zRg_j`?5A(7-G5ua)0gSfQ9h#e^(I= zb2M$LbgwUi|ojIH6{jqe1j+10WcYidwt=7L5uKu=C zD_#$y6BT4TD(92zg-+$Xp;3rofNi!AyeeIN_PlvwdEqzEy;T6+Fgha^9gzejeaMe+ zM} zKraQX`311fbSUOz-{(hSCBw!kmajq(DjGPoIx zS!goZ*2cxf<;`WiK8uB}Xg_P(Rgw^_#T~6r+5W^9AyHGGr+e|Rc+ciU9!lbg3!y!B zV0JvspKG|SuPIMlbxpM|6zC+gx{|)85r&tW0boiX`l*aaEBQ-p(?3K|8jUMaP%oK0 z-k8nDCxW`x2I``)30Hu2C$9Q>2UP@DthnBjVO>3S91c zOF;ks!@az}b_LmqhddRpe2y(ANPoH)PMF**)UbW1mcA5}a!2sNY{K-7iBp`MCrt=d zyc-t_zVQf2&6;w$9D)e7JRM~BDx{{U%+mwvz>*oy_1Iz9>&TL%o|Ua^NHM7`$t5UXRPIE$+jh;!=!*0$MrDH( zBN5*c>G%ZKTO?Go=04*FG1SUA(y~1yF51Pg&|^Gz*dQ9Ux*u`KU=0~@#g8~7pHMNG zvdM(<+QK4DRhxhiTj-WTt7Al@SgTt0ozP0#2owcX1&j^WA+b1A-jR=tk~Bffn^NyzJ0QuH}N_FV?O)_j94M>)&r7J&}gPG>4Bc4I!^pu5?&iCDS|~8>dK9-|R%X9A^PY4ta@0ZIBsGp0*~@RuRz+WUp3E2Y4oLYB z9S2+5sb|`@bugPqKUxdhs;q~h0`EteCXGK@yP6QZQFnwZvO=W?uEKSXxn7y;9yx_B z@4DC1#dUAjcGtb^a;|%|om}^NcXr)-LwDD`zHP31*Id$dk9=Av;4oy)$$97$UYaPb z$Q9EyH&%a=e5~zIKVGYz>c=_CZO8iY+CbM#S#1fj`lc@?C}%8_Jg7}b4o0f*YH59< zmhO@2=j0j;)-XgYq=4%1mq?F6J7*R^IkKdqiCbK0l@$rGeYIxJ6^Svn{hWM4_vMqd zstvSXuQDx@YU$-wI|tc zp~{|Hr@#HgCIqlr%=kr`pcvy0m&e*KyCbozHKzH6Nc&j+H*w*5j+7k!?u#RmUQ~ z+qN7^fKQ}fj5z0db|i{AW;wQAzoQ`~};mV499n9d!?nb`^kcw0#9;W4af-=3hs z#vP}YT6Aoz*6W=xTnxI$PlqNkhgdm|w(Fl?eiq$2PY<}S1A`E6EPXSb4TeNOJqfN@ z}}~FyZ%+-Mg3pT z*J}y?S+erMr^rZ(iaDX$D91@gSve4VyX~@lSEQi0Y~R(A1BDNl z?Yozs$00kh%49?qU@_gvXDF@an<`y@@M^_y2IV@ClAa9CofCPP&~fjLe6$5tTOVf*G<#tG&oy zuhv`T4k>>iHj+l0&Fwh&n%*k*$IE;h3Ldb@U)y!^W7~|p{bj1|K3l*65EBmj>(bP( zE~K?yfjvt6=DX7LB^eryX)1SUQy4*!Jf2op0(D-n@x+eyOFi z8RNR#OxwPbk7mhOd0TGw#y9u&jf0)ASF#9YOGP)|a#;-C=un_M>;kSab1);g$)2L7 zHskEMr@3*T=c*fYTP8%{K+NA6DFLE4`P`H$bpKSdu~6tUdK03SWdY=g@%++Y5S|hS zR9iO<^%beGjb7mA;wp-jR=A3=(vh~txAj(A-^OM{G!mNYQPy`y z?_Isg9enJ$vp2c3+g?pgkOya8&L9rcn)!B+zN_1dd)3mmsgSq()BQ_4gjUQi0$GQ9 z-5gkK1~4~d4b1F0lwxw2Z~8W2a>QA#brWgAbtT2yBAXr7%!iVRwxxec%v!Ngp8{gG zEq$`B`9QTbmv39+`#`VmBGZJ%#>dQJ z>yz|=<}wp&JAV&atOg}9ShS6J;rV2_Z3h=ScTg?a!N=O#J>P+pp0`$;ZLPLCd)xBf zwt6p<`C#4)RRHKlO)A+2rniS{2PSwhrv2*{CtTzNEKj)%qgM4P8%3e8gg3)dM3hsd_{E=s?}^LM<7MFT^=r zlcgWt_S%fQR(HJJo170wy6|~M`hp#j)R&R#CH+F^DY$*P_$p?wlB%Yo+di^;p3oXj zlDuvGaTDRPkc#k1{5w)$#fO^+K#s*bow8kwK+>^&Kx=mlP2S>ugsoeImXPoN3oPUis-{&-BGp za!JjS8*0U}q6=$B`!^-K&9971C$%*CbQ$B`(e(Oc@W}ciiQ>s1QcGV9Q|ape)$wj# z@u)vsOTQmpF8vO+y>WSe`tjhQUm8uaWU}mexp2VPspo2oil_d*?wVk zT4xYIb^iC3gH&hNc8QCfskVsoXOLTyV|OdJClcR$AxLFdPSfYZrZ2>wX9~0CwmtgA zWH{OA?4|9xE!NUM+_}R`-o26JkRy{a$$f@NCw(^b=h&poEY*3NZcmbpfF#rK@2Ne3 z0;FlylQnw|LCf0yvOmgY_*o#OcIi`u`FtHWF}aBI2`jC@c~PYt_~HRkL}JyG(ZXgR zutU2sN|3@!`Lt>URy+ku_h?nA9h2Z87CQFqOt>gyHv^;_NgjrfvHD zFNW!>gkU~|CEBb~`=Tpa>@2+f`5Xnq+@o$JvsZ%4nv107p04RP3*$;1rJ5El*?~i; zjgCFhb|788WFJP6yrvo0s3Dp_w%+gnRrvX4tDYPv&O=%+>wS=&yepNaC%AuF0ky(B z+_s#22)|*>`pQ2Mnk$$Mt=~nBMR1h8ysw@Nh8VD9)I=46iraR&I!!$Cf^Rvp7N)&= zj>dyc&1ahzEfIygSIpLvApjo=Xd9DU!LF?3Y8np)67{%hAN!UtMX%(LCLc@Lf)A7F8%{#rh{p<6^-YP&Vp zFDw$$L2@DL3%Il7snxd;9t^|+B#pQQBb>WU-0w^ygt{-TjDRK4``R1z{KzWD%e`c1 z7O9h#^TRm{(jHk|2#vp=zg5}@lwaJbra>!Z#-EVd?6_va=TjQF^bIj~ze zktKG5W zMg>$)9`ZYlP`BB%G=Vl-0)z9}A;Vyf1MXg3=YWPC(o9RTZx#Ta6SI{sF5sY&%cAez zxdNf%SoK+iAFjPHs|PE^OnJ$MvI7LQ**D7zIH!cNHc{U@%M)ldCRrn#f6V9QexaPjf3t=P%Ea!oDbw zj7RaxeD04FNJUjlmK>{Ds+?G?7@CJP1Z?FRARw|eZrj+JW}>0`FJfTW8ldP>zF3hKPKU0DdZX<2ea(e%9@*x7nc?q1GxDAeTKr(W3cdNNxMIZerxC5nqxN9{Xv1)~NwMV@;gIy0JT zyK$4Vpaz4@+1PKK%dsw#fhQ4Jfp?LG7@|I*BXmx{DUR5hHT|aj)t=fOa~y#&Bxgr) z91rQ`m-1O*8|Zu6MK&lx4iA%ys|h03SVDUCa~>4An8 zkKm=yecXr`AvTT2R~P5YVEti8UwEyaqB{Cisci*Y(y0%w8S~Sa0q+l>GENY%`wtlm7>s zA%h0aF90+14fNhPMcatd^TXgY{;r4(6qBouHtPxgAXbTCYu95n(6XyyiVvJua{kHx z(8fLgZ$Ee8xBtaG|L6a#hyU{J|HJub-Z1+cH9o&<@uScGll4D3r_XPG_K%<6Q~Q(u zNp0SJ;n#leAHQ-(U7vscfBVq)-}%&czowZV`hO38DgCn#{yO@uzU8J@SDHT+_{?4E3^Os+K&kMisZ~tDuKL7Vm%>2PO zKXmZ#=<^N#%f}A=+T`E=>+16@zkd4%{e&;XT{LCLe@bZ75&!l@GsOUX8<|I1?<13B3A2Eamdm(=?*tFB9?;wgqDNMfE zcT2L{;S#toT`ypv-vJRdaD)Rwvfk>+odI3JbS*Vt_idy%w_10M!X+S3zFD?gRQnrS zsvK<D!(OTm&=B2+;#Fk<1>HDcFF?^q^9FIz7%YN0aWaAw9USukU&R2ehX@O1> zaIKU=qXeEhrWRrg3okDrvvb)5CN27;fEvmKCzg6d36YVt3K6?*zEfjJf77g=m7-RU zlb+S(2m4ZJI|Osr_k>4J?aFS_WQ=K3o7^=A1cNS}TP457`Q?4kEd5}NSvY{WzeCOX zIb;HrYrcJ7z07@(Wyrd8$~(mjtGj7jLzII;XLT>-{J~v)4WE%?O>1gnBL-H^*12Gl zkwOULEUW;gU#Q4T=#p($Rd2Dr#BuTR7pf>&ygbihfTTDzF@!GjLaJA+L}h(Aa2t}6 z2Q)W*P{*)~9j5|q4=Rr#@7vN*5luhW{F%ymN<4!*T_1-1TQv51u|*C=0EruC*$@Rq z>+?%U!5~%s_NvZEYooEemX;s;`cJH0NRy#lE%cW;w-6=Ewq)7yp&@w`3*(D$E_e+z zU4=66hlL77yR27jIOFDxYjKjSSU|RtdmK7~k{H|Jyg$!lJGN++~M9EK~a6H#r zEtE%IGu(1Q3Pjn=Vld8F$p_|}b>thwSh{02l5Fvh;ZlYB9r%|fl}ddYCG!$-#4KsO zNyva_#5ZrTzx}2IH_~R1wU>&m8i`1+i?l zShOZE#YZ@I3fu{xY&y)5|K=A-za*7}KtnrtOU%o8i*mN9uL3<;XF_j*cL5(jUK=PJ zkvVuQZ~az1e+YjZR=&zj7n{pJ6zq-{)FhU3R+^IogdOmhLn~hZfHHw$aSJ=XH z4EfmFad>pDK=UFnwM0ImaJ3KPz7io3Oc({D5d5#s*W!9O)WF`$*&VJFkJjJZHZc$%aFHF%a9lrN%oxiEY4FyZ>pZ)5llRiJikJ9%I7e8_){vl>AaD*O z2-l>Le7{kUx*(d>lRHv7*Rx3YD#Hl9B_$jR1(JW^&XkAp-W*E)@NC;vnAR z_t|)t-wZszS?`B)D08E=T=gdWL7f+7d0Nv8hBbJ<+7R0LA+$P&Bj+Dpa7(E!p3+8Foyx) zSWEB7mI1ar+_eI_gh_BVdnMOuHc@*yg2xgptfGX>!NsKMMyPBaW;`sIgmG1z74RTt zfR`_j15=A$Y%CLguFtl~n=VNl$pGo}fv6;N*iNu5O;S zqc_!!RJjIW&bVpQ)>oInun_?mM*eMG!1uJF)~Fhc*PnKv^%T@`&V55T%#f;JJX7!%kG(iB-5FxEABH*xHu{N)U-j)3x}{Vw{C|=yVIZ1HcFvr2iELG8J!N=oecM zh6*P?B>mLN*1mc^i7LH%p}@2Aq^3KksR|;fEROa_> zSD`q2Zaf@zAT2KzEn(Oq$BkyUy6V+AL*8KQU2xllvr$@p>>oB)uM=i~NRtaGEKdrJ4>ooISu9ANr5FxEPEz8<@ZN;lhU&(Wd7a2gR1 z7OaIbpXniXpJZ5meV}utM^BS)_dFRVxbweA?BS&%E4!ssXJXo5jBFcV%KWGXNMKVV zYSxd|(0E$$Q=98j0|gnaD(YAZ$r5M55?CJFJH0o>cm%jw2`^!|m;zs?Dc2NU2wMFL z%h7Fm0J6yoGS8tz4sc?;mWou%+UKzW5Fc|Ww!1B9o2w2DQlV~mhYhpPG7*>G+IB$f zt^J`ODZGGCqy;6^9_MGc0|XrqKJo4-!n7I-0Bk@c2Drxe=r#lcpNI<>5J)h?oVX92 zHn&TLJC*(P&4qeJnqRCoKe;klB%tZi_R03P_qo0&==;009}~HGbP??cl7NCLh%321OLi9w4HC`M*+rLA1AZ~)y2{8L zh)l{BH@o$bY}T6t;IY+~pubh$u|n7lS%!7rgkj#YiaxGoGUk?xHtVIx3U4B=jOniY zFE3kXF znbwI3ONQn6{x(+(hciX7(&=TGc0StwL;)_rH2=03&~n%-prpx`!PQdHP-ilYJ%EZQ zAS)%ZOp_Z@!QP(G2VpNgJ?|ou6qxvCFfptWdX-I_jUtH|`Ne$v{?pTucY{^u7)v6y z-jw`Y6k!oTVkfUjF=8oH&gmZ#g0wy)iq9vcjWVJLr-Eh&w1;wKZUag#=ZC-{D)IAs z3@^`mF(>1^&n>N)RsCqJ6am%vE@&BwaMNw`A~B~zft57*(r3bkU&TL% z-2HR12xm(vkHayl;Ij9BM2<&Z$XuvYZWw>{g$ag*)ob;mciV&tSf2&PEZjZ;= z2UrkT2HoAq@3hI+VKfx@*Lux&PCh~yWvE$*X(4|rgrBzxcY-isRG`Ln;X3Cf=w+nc zIx78Nk_>S*nAMaKdDFs5YFLJRX>>^uTVdfmv28GB>7dQZHqSPVNw7jxSn8FuY;aK0 zD;M0Ly*aBNXEs&|ELfZ64iw#+5o8`IgQzW>j!=hbqhRG=hU|LsLXK)wntC7x8F<4* zj{E`a6S#OprqfK2@t~CPzn)B0)_f}D;!n}u5zqr170igv5R9st46>MA{+QCwCh1p2y5FY@ ze=UqAFm6uPY4NFluK;GV1?YnuddJ=mazvXZppWG=I7cayMWljW1Mh?L^`1lbA@#tG zmmPFIQJ2qLahJ((oM4#F$FoluvQBSIlQacoq@UDM{#w%A{X6aI6y8S^@_LK)B_l%M ze`t#{32j9PHE24D%cyOJxilS_0Ed>mTu)xkXH{PZ9IN;(fHGhm{C~=K;X!#t6mL6;ov*ZFFbnsX0MLuRU@#slr6eQ3pq#T9I3Jac1f`NY z%z-Mx%awQiTAQPdGylMLbh21jUscVQCh#f~tQpIx+)~fW0>}Xqos;J+lba(fO*e+s z$?*C3Vk4imXEQvqC8F3)Z|00@yOKVsDcLa?SX>U|a7Xs?=WNqWH3HSFco~2j6k*50 z<32Z~DLl7|kt13Jq%boH%Xt{uvUPGK)1@(m~nh!3WAI2*6epI z#oQxG+n>Oka~0)`CL-zYNu)s8G1T2NocIRoJX8|8Vu^6(oUJa+C6^f3!bQZO1j+qJ zwalt@c?89uO6juNAHiDFPjTG_-k9I;>%Vu${A>U4#PIQ7`i(lpQ6i{O`+0`;lssp<)u0@FT38+KtR zlby@cv|nML(5pC&VQ7tO3ZzHTvP*Tm6^)6=8~6Pq{iu8PAdv%s4h1kyi0qttHjKQ3Y%s&fJV)zr|lt2aD3@yg%) z&42XI@-wdw21t1|Lt3cLTJ&wbiFUAM+UF8pNMf)!Vk|;A9WW%=r9F}-*4sE8j0Hml z|BnGMm4E*`tN7p46HvCQSQpJKu=&l4h0E~8s7(|8M@t*%c%cYCaPMYwld8*RS_-iQ zeNqVrzejCkH$#DvP^o--3Vf|r@?f*u(n%)n(UtqHE* z=OZ~2^#)Y5?=C@oGY`!rW}5C#6Ela_xD*-hrJZem%!JNi^)Ooo20y+2uUt!WHrQpM zN2QrquNRvt)-s%j-oxj0e9m$SsI_l z5_Xgp(;<-^L`SS*bSRCcH*F)lEHlD^ByccAhx9>*#Xuyd(%xeTkefPGPij<@fmmnY z1VjcIz!>9M47*`bGj2beg~Cd@#r*=u1B!b|^~Ea66NV>{)TJ{KPmp~CaQuiJO7#G2 zZ-ZL4m?y9Dnn`hb_4SO+icWAQ6Bt*aS4GiWhsO5k7-o4<68Yr(Y<{afi6mF8mAJ(f zflJC_PFtUw*cCX0;IV?2p_~VNYG!HE;*1eE?I$}aCzJ)euV(Wh*66B5jK%o8iy*-_ za2ddla3Nsjh$82(HY)o7TNi|1-2!=WsfAhC04fW2sOPius}!N8{)MU~w`S?jIA&bB zyi?hyy*r2O;6D@iv|t}B$~58RySjNU_@gT`$)N&BU8sVkxfh&q4aSHewuOvK5|gh> z55a-9{Sh*xEe|3p*MJ*wzKJ%}0ELaRBmVl1d~ZR)rXTI6!KdARsi&aOB46@8l-vMsftxRN5_|5G3Q#-3IS- zIA;%|9=0}U4Qiv`5Y!!)E}CovVnA7Gp%pJSpyomRUUg% zd6Q$rA`Tglei!KOIg?&jV<5ypJ%k^Ns!9nyezukm;gU0;ut-$65J?j<`>+U%<$(5| z;Y+N#NN$VvY9#7T7p86EBGqje-zt_UN&y)qfBa{2R6ma&u=k)`=57ZES?c1;+izsj z9o$B4zvJhG%E{oVLB+mU504+%--a&I`vJgf;i;vclrNDhXrDOr#^i~Wzq^sX=?meH zY8M6|#(u(3I31x+PCF>$OFxD}E#gRqG}Y0AT0jVZono{IRr9Ap$=lHZ;{1}`a5)0@ zI8cSh31H}zYRm?!%`|cou$ho6P;UFnnRt*A%-F3tmkU4?yXJSV^%g*j66?#2b%NYr zK!}(K33|W^=lKrKFb3Yb)HO5;`%h1d zvo|+%?2S88nXv^bseUV)gX_9SGN=AesV3)&r^$`F-fOnGkqwOsa4|ycRI)9{P}SW9 zj42R|Zx%Gt2=3y1GA1G`t**RI%kjS1<|3feD6fR0G83{*d=fxaY4Q$UEx_(Jua+Yd z6L&vk6Z!Ysrv;{}yH88GK|iJ*EnjK#XelW5X}Moq(zRuOFn=4hP2p2fHhra?_QDWw z<$VjptGF8`d|Ggf_!;;B@v!wpd_$#&fYH~@GS$maPIteXojhr(SJ@~~P|TBx zQf_QN@yZD?nL1VIHuwvx`H#O>&ds%?Z_g2xJ(`@S6506zoo;RS$NvlR-MP+L4-_tt zQ48&Ieal~@2ha6D|MBJZRR7nXDm}G=!l)DwHZyTPek3}){B^P#85FUj?7%|W_O0)b{nkdk_tjo(leaGOg~)|*Q{$Nq zA#w?h6C-LeR0pGLgLSYAS2m7}gP zoxCc&S65Ze2>jh&byZH1-uNTc4E(v(0z@i?lt5|txBt@4GfDDwcodWb3PU=ouoz1; zO#q@jzP!ymeHG>)NpC|o*n1x#Pix8>@WnKnQg)J+78)H`jqo_h(OMk6{kSJBO3RM} zPjsmPceE<;XJAa!{kd zxMG%u3Y&)RHB@w186gj47NlRm`sA(RZ|tK|dZ@G$-tusk-$Kg0X_p<=O&~8F)7;`x z8b9pWdn#5-OOrb#FuGaV)Fy`7MdO0?mNlcD!(S7{cYN*Sf+*uy&YNFAkGUa#o!BlY zB)rm^dNkvZ6;Tfa1O3ENjw^KXQ50XYJpq?3G&%ti>^xCEnumgRcx}@>Q^m52AnKUy zb1|;jbfHd`&6-{2kREneKX|aM?=#jx#gEc=U6pE}4urA-2Su;YWs9C%T(}moi(M&} z>8{Htf%H^*4bwnq#)4#3!aLi<^u*5`>UrX@^AqqDgXGF(-nM;rY3DzkD|;@RUWN_C zA;QK<7az=gaQN{E?CQmfmmS0FAXXGaNx#s*3>xcE0}C?F$Y8=uBhe1x^2p!*-PWHIcj!Kzp!GCa#aMT*NtY2F8psX8vLo)F3Fa1$W7x15EC~>sN zs6q?6I=f9#5f^kR?OshlxGpAu6?Lu*d>G_g>}#HRoLbOxc`rqu8wwnz_&Kh0e%Fo{ z*AZ8`?T`A(qnQv4co^=sli?=g;{uomj1l6X`9_}F_T6_Sd3;~#h~ps`2+6Olc@dPW z-MqFZN%w&o(#ab5VIO%VJ5#l#U7q~}XW&3m`d>jO&2%k-rR_ar|m=#w|&5 z$z+M?C4c?P#3a|V$F~WS+$&6SE10A+lVaNCS+>V3p_z0&c0UM^K5n41JFvza1Ib4T zb<2zPjSlqCLf)!E?WL?9BeyFK-e;}KUzE#>>6GdeeD|$-+IUF+KRR&eh7{5QZ{`ND z7^H=zfRhZx2an1$aRvB)7@zuL&|(#pY?A4vf8O&*;W-^>M&VigX~@J^a`)4Zwl`x~ zsD;`vMP_&-GKyj-j=GOz$8&^4vW%WLfIj(?b6y=gR`USg0Zi$OXG%8zCl zzj}!gu6RXX+P0!%vM`{*d%xQjJ`#y|6cPe~Pm`u7_-C;lMd=~1KW?2|z$4Ags@t|M zd9oO}7`W)8z4Px}zVo85&Q)B2gLZl&x9?7FU%{$0E`}+L4*V3*fzz4p-tof^R_>p# zv-ETc3+#%{w)f|rqL=Qk%bL)c3)hU0?8oD3h(f~R2G8G0JD&8-Pb)GTRSfLEq%+vb z)9$*JB+EKAj9nJHExxa86Gg~moJtF!`|w_xJXXUmie&SL(|mlS_&L5J(N5Y#CA%;3 zd{>gR4{Q5o$QlAOhPh(C^3%ST^80am$fpZ_+& zxu<2NU%#_w;E+8%O`@9+sB>4(=%_uLz8Iby?it;0Pl8=+qG!xokPr@UcwA~Hd&Y)h z?{m$2dj^K>o6Al2_Y92LR}r?m7ngv=hk6DM*rNzJXL#VPQ9miyU885<{AjepDI&4C z-al-iiRPS2cKElG;%V?*wV@`88w**C$(6!eE{)?BATQ}g{!6D%o)G+;wO~ef50eHb zK)eTKm;fY2k88d6#+z18Fz-yd{=E-m^+Ltb^rvV}h?s&;}d*`s%G z*(CWYPck0Emzt}Zxjv(uvv3C4n-mVp+=6?HX3I6PXl-7iTV#2!>RP_ZY~&7v3+)_U z3le+_-ypteta6U7Ojy8i7K;IQq^Z!71JVe%L!69$X9zjKZh#A!G(~iP*xhAVv+B&b zAwpiUJFa*U|0+gXS$(8DikNyu>U88>eD>rYi;_H}YSFuE$0_Mk`p8y~mN*R4X~_c| zjwEOs1U4HZ9O(|bUeHddaR%`epGO(bZcZ-PXRSX(4EPDIcuvB?&?jHP3Vd~+M1Zr& z9nap=RK+<5!Jo5LoQOQ_d`Y&iVZY;s&F-);OCfWTIeWU_0w{xbW4sjI=NmVTqk-nl zWAfNx_xxl^%c1GQw1>k2w-{9=yia;Eqp+rf%E(1_vq?RoODP} z=Vey>1YTBF>dj+_kl}-c? z$9XacFUfTP`dYME@qh0 z4XB_{a!=q(yUGpQ@5eG@{zS=pYXh9NA4p|K%M%VPW|>6PU}fj~<-Ma0Vv*AzobbK$ zS+Fw;z6H3M;XR1UrGg(Go^2-q2M?CGfRsdG{#?}5)AtHei|%!*1!h8%sDq5hr_VMj zciUX#>j@6*hfhsPkuFexH*w}*C$N|Lw$dY7<)bLi`u7kZr#y(s)-KfCaTC=r{1A^h zUhw2V&;fcH^K7?Ke#!)A^CVrj1m`x6Qv~~w(3LV1Rd}r^VfMjVc389u4%EmFU}7)% zhH4@~S5C%l%sL`0jt{D<33^Gn?k{6TU~_Krqnp(t@sD=jIx>_k@){Jt;{vhgfW=Uv z7Y&MtpT&sA>bsL;F*(A0-VJqohmPrBb>p@1fy5swzTddOLji0#w+A^BMNi|548Jtp=i)E4;-C|r%Qo= zQ+kZ0L);28ai`&I7>`8QwYf$7&IdmghYxSBMFzh`HRRAkb<#3JbX&%6#m2r2#auLJ z6uWBO*aE`{+kQ%RC$kn2?Z&#v@l)lN57@dw*lAqXD6!*Jpp9DIZNTeq4d=Hx_ftyLj!i!7Z;YB}T40K-7%Rreb5nt&x z%l1xfV`aGvIvjH{O=j&dNvl0WIBJ907RJ(7YRJ`b?Us03Etpnpk+<;Hu-qZKMZN_| z_J?`cQo|mGXJrej?s}#E<|${;*3V>E>7xx5_|HhGfkQwr`OPfP%}`p#4P?pp(mq6* zI5UlH#Wv*P1(3VG_OuMbkHRTs-Qb^KRdE-UbjXMd7&Y=NR0 zOp0wVHoWM&+)B7zQX&{>o;@DUhP?E`?duH$4uc}!0$GOr@a*KZjq45#bqb9iQF?$Q zF0BoW0Rl@~W#aE_FFr3`L*$uVzgrtv$`7l7ksT*V#qX8~IG(7-GkCRm?I%nRngQec zLV8yCzD3y;Ma$z5V%oVFN(-ecJLNw+B?M&QJ@Ma)xj>zUCeO$h?rsZ?w^>ExXKw+8 zi{T=dYiecmEmzQ$jW5?ewo8Rl#IP6%j6alfA|c&08&>>a_{HFhWJFue4uz}CX?gyp zF~dJ)S>nEeKhWD>u6?7X)KMl&GiLx^M-oSXdxHn!7A{gX<*J9y9t$v0J=PAm2ppD6 zx52X?CORY)W@z$D3P~8_={9uV&%xwD>hZ!MPs_zb=Ks|dG*O$)?3B$|G_FD%`v*P6 z#Pb_}033i^xLpSfr42VhMcY+rO&+6lTlCOH4wxSGGqFt=57>I3k~EP2cIT!6q0oKyTRIw=yiA zD8zZYNro+g2@(|@M=0Jpu+g?l%L?U0I5l?!_6T%WSx4)+5bc5VVT%`251E(F1^c%r zp0!-tU>_E(eVpwM5LK8HC>ehaWfwG=n6Uj2AxPv)=}*R^*Wv@PbIoNVtp&K+?Fr`L>~W8i;*n>hm`9*sJB-6&eYrZ3lv-EyQU#cWSNN4;+o#WgE^`ZUM@K5e30^HOg>w znN)1nCw399=or>vGJ={j6_ute@g-wzR>CU<X}{EMF~TXZKJLO%%5nSP%{xJ9hq z>REXlaTi?$3P0@hN}-s(7`^BVnk5q?_mU$k>1qK~VUujw(7gdD_Z>YuVZ4kXNKxsc zrR%|J7RO_=8iM?4A82}ZeYG)8Td61Yz<{Il&aNGVnIOYLaBm0-G!Z}sQ`Uy%>1M-DwtMz z%?;&&jD!2ML|UdfrC%-mdTm3v%V~r&06EKp)zKHl?GssQR7*A2BVQ zJI;hC06_2tkUIe#Q!K$Qz7rvCXE5FwikV{B;m&Y;6;KR8>I%nyE-1!^5V`2TDp35| zHYnay1g5joR|n0nbjPMx8?FMHZS+8zE(6sdq)w>r^8RH|>o<1>%FBWzQ<&z`lt}9* zsCBXRqd=`|m!fHXgV29edqevple5LQ$I|P1hoLU_1ZoKJ4oD2b3P$nd+eWi?e5XPBLbE})M&_UhJ`>Y>3hBU~;JSp7=nh+}(-JPoF4r$UVO*+L(&7s;KW(2yD>4nhj& zQmBtj!a}q^w}K*CoH~10z9be%qLXT0t~WhRo5r(q28msDWdUFa5TnVqpQid#WOjtA0FM1Wy<>hr((%;lus5xzd5lAc?P-^@bPpCIgJS<=cL8+GP>ur~7f|$g; zRA8UMs+3&5Xjd0uzXm_&ISYd}qohwOflB*Nccmk}eTyajvk{#3 z3EzPyWR)|0MO9=&qa`Ex4xQ$AzH!vtkGHLFbmxUn^!72iPQ+hm(`Gw*v*>ZK zjw#>`v5206>yH^`9_26{*DZrkv?dMA5#e6>5VXKb*5Nh6`VNPZpaJ)HxugsEb2af?-;x(AGr(fW~>7x-sSpa{!?38c+T9 zOS4p8c4d(ivZiT>#DgAhQ>xY7}s z6P*CuYM*GZS$}&Ljgr;fHJwv#(#&SVcN@*NK)$c*tWo8XJ#b^Ez|KksrZ(HR1YMezY>tVm9AM8U}?AJJ~V z9GDVL)ut~pAGL)(1;I;kqV?pj#8FXLq^o7#?uIlZ8|eCmkBF8z?(Lf@$b*Q9-?Ic7 z63vE=GNWcK#-4SF_4WOxB5NnzD0}^URr#cbKVu5W`Jj0!=3=DE?iA$t1JPb6C4sK7 zk8XvVNl<2mn^C0=pLTBWt;F~CVd)JBFK}y?KdW!I>3aqTTBO%Jo*wVr1D_J{nW1jO zQ<1HNNhArFyen4Mq;qEjrL551)D??t4W3u67ClUy%66lyPt=bAc)`D1>qQAC4FJ^S&VK{F)Z!|5XDL%N}fL}6z7vq zBDJFCDPIw6BrMDa89H+7Z6rpy_4eVsUaz}?GW>Le;mIt!@ENh=(Y9CEEUFVQpU^Pr zk=yFYr`y#SRaxc=wZT(em2t>FSFFBhFI835|NdF0s-il>KofL&KT-c8SW*7$$6x*X zL+B_T{E3Rf(D{#}isJh_Xehvd;G!KB6js0O-%ItxXcYU!pW$ey%AjZq3Nddxt@D&t z_%S4V;lp7EJrYQ!U%I!{ zIhVzZ=h0{t?BLQCO#4`f7a}NSt_e4k9S9*hGsx3(&1_HjnvD`1Qw5Lk9Fp`b^`Ecr9}O((Bx%>R`P1c{biUa%lgTJiMp;5I z&W41FP)jb`FIj{yGn*8pp|Bjmn{jspMQuAn+UU^QYT?M$di%z}Frr|3JW3&#dOagQ z5(V-!<6${Y$ZZ%;(xlm2_%yVuD$TpX8-+kzWoxd^Kt<&=bZ5&f*$BH79E?;Cff!f| z$zTe*ck+C_E<^trcVs6ASCcC75da4o%mhVU@ zI7Z+30!<-zEuo)5wnQbPsHGdSv2y0N&=(%-&8UzVyOA9v&Ncg5+tF|hsa;`ef0JLL zSXCScF?sPk`c$Y0C6N5U!tYFiQ|f;*xPRa_-6b5-e1ZGM3s+i_Q=gU4_}sX+_TjD7 zM7G(R6B_OQ6tP#ofZ#umzQRkY97b-KJ|!8LgoJ^0MGQTBpH0O(AIF<-Lg7|tr0~Ka z!i+_se_4aX(pq-g@$G{j{ImR)nBLokbq&&XgWBX&h=OitF|9{<-IVt3{AQFyX50t$ zJfcA-P3x1IeS;H${ugu(PDaG zZqiG)EW$AdyC24oV~Yq_t!cGF??JF zhiY%-&JT~Y1H`FH9PiNxHGheak@UaNb{9Ez1bor4Vgp=lW7jkJIF%=*7bt9ApwwX; z)E`C30N?<1!Q5sF-U@7U1f~J7q^dgTpjax90*p_c3et*Pz#2H4L&T%5ENk_^mnN5u7nGy}=|dn?-!$@_GdWCO{V`k>lO3`9z9%t+sT nPrMO9(OAhfkYw#BK$qWGX=9hr`!<(L@wSEQ=Vs3bdz1eU{r?zF literal 0 HcmV?d00001 diff --git a/node/Cargo.toml b/node/Cargo.toml index 0f9a52fa0..228e5a025 100644 --- a/node/Cargo.toml +++ b/node/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "midnight-node" -version = "2.0.0" +version = "2.1.0" description = "Midnight blockchain node" authors = ["Substrate DevHub "] homepage = "https://substrate.io/" diff --git a/runtime/src/lib.rs b/runtime/src/lib.rs index c6f7facc0..e54dd7b5d 100644 --- a/runtime/src/lib.rs +++ b/runtime/src/lib.rs @@ -280,7 +280,7 @@ pub const VERSION: RuntimeVersion = RuntimeVersion { // The version of the runtime specification. A full node will not attempt to use its native // runtime in substitute for the on-chain Wasm runtime unless all of `spec_name`, // `spec_version`, and `authoring_version` are the same between Wasm and native. - spec_version: 002_000_000, + spec_version: 002_001_000, impl_version: 0, apis: RUNTIME_API_VERSIONS, transaction_version: 4, From f6b1d1f80119329ac6006eb65ca05c494428ce19 Mon Sep 17 00:00:00 2001 From: Oscar Bailey <79094698+ozgb@users.noreply.github.com> Date: Wed, 12 Aug 2026 16:09:55 +0100 Subject: [PATCH 08/13] chore(toolkit): add support for runtime 2.1.0 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> --- metadata/src/lib.rs | 5 ++++- util/toolkit/src/fetcher/compute_task.rs | 11 ++++++++++- util/toolkit/src/fetcher/runtimes.rs | 9 +++++++++ 3 files changed, 23 insertions(+), 2 deletions(-) diff --git a/metadata/src/lib.rs b/metadata/src/lib.rs index 4c59f48a4..30a73e909 100644 --- a/metadata/src/lib.rs +++ b/metadata/src/lib.rs @@ -11,4 +11,7 @@ pub mod midnight_metadata_1_0_0 {} #[subxt::subxt(runtime_metadata_path = "static/midnight_metadata_2.0.0.scale")] pub mod midnight_metadata_2_0_0 {} -pub use midnight_metadata_2_0_0 as midnight_metadata_latest; +#[subxt::subxt(runtime_metadata_path = "static/midnight_metadata_2.1.0.scale")] +pub mod midnight_metadata_2_1_0 {} + +pub use midnight_metadata_2_1_0 as midnight_metadata_latest; diff --git a/util/toolkit/src/fetcher/compute_task.rs b/util/toolkit/src/fetcher/compute_task.rs index ba09a1d3d..508525ed9 100644 --- a/util/toolkit/src/fetcher/compute_task.rs +++ b/util/toolkit/src/fetcher/compute_task.rs @@ -25,7 +25,8 @@ use crate::{ fetch_storage::{FetchStorage, FetchedBlock}, runtimes::{ MidnightMetadata, MidnightMetadata0_21_0, MidnightMetadata0_22_0, - MidnightMetadata1_0_0, MidnightMetadata2_0_0, RuntimeVersion, RuntimeVersionError, + MidnightMetadata1_0_0, MidnightMetadata2_0_0, MidnightMetadata2_1_0, RuntimeVersion, + RuntimeVersionError, }, }, }; @@ -173,6 +174,14 @@ impl ComputeTask { ) .await }, + RuntimeVersion::V2_1_0 => { + Self::process_block_with_protocol::( + block, + &header, + spec_version, + ) + .await + }, } } diff --git a/util/toolkit/src/fetcher/runtimes.rs b/util/toolkit/src/fetcher/runtimes.rs index 9eae8da2c..ec4f6170a 100644 --- a/util/toolkit/src/fetcher/runtimes.rs +++ b/util/toolkit/src/fetcher/runtimes.rs @@ -26,6 +26,7 @@ pub enum RuntimeVersion { V0_22_0, V1_0_0, V2_0_0, + V2_1_0, } impl TryFrom for RuntimeVersion { type Error = RuntimeVersionError; @@ -35,6 +36,7 @@ impl TryFrom for RuntimeVersion { 000_022_000 => Ok(Self::V0_22_0), 001_000_000 => Ok(Self::V1_0_0), 002_000_000 => Ok(Self::V2_0_0), + 002_001_000 => Ok(Self::V2_1_0), _ => Err(RuntimeVersionError::UnsupportedBlockVersion(value)), } } @@ -48,6 +50,7 @@ impl RuntimeVersion { Self::V0_22_0 => 000_022_000, Self::V1_0_0 => 001_000_000, Self::V2_0_0 => 002_000_000, + Self::V2_1_0 => 002_001_000, } } @@ -157,3 +160,9 @@ impl_midnight_metadata!( mn_meta_2_0_0, midnight_node_metadata::midnight_metadata_2_0_0 ); + +impl_midnight_metadata!( + MidnightMetadata2_1_0, + mn_meta_2_1_0, + midnight_node_metadata::midnight_metadata_2_1_0 +); From 14ef576d2fca240dd777adcc2bd0281cbff8d699 Mon Sep 17 00:00:00 2001 From: Oscar Bailey <79094698+ozgb@users.noreply.github.com> Date: Thu, 13 Aug 2026 11:30:47 +0100 Subject: [PATCH 09/13] chore: npm audit fix Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> --- local-environment/package-lock.json | 48 ++++++++++++++++++++++------- util/toolkit-js/package-lock.json | 48 ++++++----------------------- 2 files changed, 47 insertions(+), 49 deletions(-) diff --git a/local-environment/package-lock.json b/local-environment/package-lock.json index 57fd4af71..b7b36a78d 100644 --- a/local-environment/package-lock.json +++ b/local-environment/package-lock.json @@ -1450,6 +1450,18 @@ "node": ">=0.4.0" } }, + "node_modules/agent-base": { + "version": "6.0.2", + "resolved": "https://registry.npmjs.org/agent-base/-/agent-base-6.0.2.tgz", + "integrity": "sha512-RZNwNclF7+MS/8bDg70amg32dyeZGZxiDuQmZxKLAlQjr3jGyLx+4Kkk58UO7D2QdgFIQCovuSuZESne6RG6XQ==", + "license": "MIT", + "dependencies": { + "debug": "4" + }, + "engines": { + "node": ">= 6.0.0" + } + }, "node_modules/ajv": { "version": "6.14.0", "resolved": "https://registry.npmjs.org/ajv/-/ajv-6.14.0.tgz", @@ -1513,13 +1525,14 @@ "license": "MIT" }, "node_modules/axios": { - "version": "1.16.0", - "resolved": "https://registry.npmjs.org/axios/-/axios-1.16.0.tgz", - "integrity": "sha512-6hp5CwvTPlN2A31g5dxnwAX0orzM7pmCRDLnZSX772mv8WDqICwFjowHuPs04Mc8deIld1+ejhtaMn5vp6b+1w==", + "version": "1.19.0", + "resolved": "https://registry.npmjs.org/axios/-/axios-1.19.0.tgz", + "integrity": "sha512-ht/iuYZXEjFxLH/Hkezgd7m6JKlHHXEUSneaDz8uZe1Gj5QZtCnpyDsckvAiEnT89OEbCLmnte4R4sn7P0EKFw==", "license": "MIT", "dependencies": { "follow-redirects": "^1.16.0", - "form-data": "^4.0.5", + "form-data": "^4.0.6", + "https-proxy-agent": "^5.0.1", "proxy-from-env": "^2.1.0" } }, @@ -1539,15 +1552,15 @@ "license": "MIT" }, "node_modules/brace-expansion": { - "version": "5.0.6", - "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-5.0.6.tgz", - "integrity": "sha512-kLpxurY4Z4r9sgMsyG0Z9uzsBlgiU/EFKhj/h91/8yHu0edo7XuixOIH3VcJ8kkxs6/jPzoI6U9Vj3WqbMQ94g==", + "version": "5.0.9", + "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-5.0.9.tgz", + "integrity": "sha512-ScQ4IuvIEF1TMlP7Zt+vjJ//9zlPb2SDcxWxM3bk8s6t6GGdJ7KO1dCcTidOPJKePW30LE/2cT7wCyPho9/Wxg==", "license": "MIT", "dependencies": { "balanced-match": "^4.0.2" }, "engines": { - "node": "18 || 20 || >=22" + "node": "20 || >=22" } }, "node_modules/call-bind-apply-helpers": { @@ -2268,6 +2281,19 @@ "node": ">= 0.4" } }, + "node_modules/https-proxy-agent": { + "version": "5.0.1", + "resolved": "https://registry.npmjs.org/https-proxy-agent/-/https-proxy-agent-5.0.1.tgz", + "integrity": "sha512-dFcAjpTQFgoLMzC2VwU+C/CbS7uRL0lWmxDITmqm7C+7F0Odmj6s9l6alZc6AELXhrnggM2CeWSXHGOdX2YtwA==", + "license": "MIT", + "dependencies": { + "agent-base": "6", + "debug": "4" + }, + "engines": { + "node": ">= 6" + } + }, "node_modules/ignore": { "version": "5.3.2", "resolved": "https://registry.npmjs.org/ignore/-/ignore-5.3.2.tgz", @@ -2372,9 +2398,9 @@ } }, "node_modules/js-yaml": { - "version": "4.2.0", - "resolved": "https://registry.npmjs.org/js-yaml/-/js-yaml-4.2.0.tgz", - "integrity": "sha512-ePWsvanv0DWuDRsW8dnt+R4jQ31SCRCQ7hhNcPXZPsoBZiemuZNYGf7adZdqX2D86j6rvKp3RpCxVTSb8WQlOw==", + "version": "4.3.1", + "resolved": "https://registry.npmjs.org/js-yaml/-/js-yaml-4.3.1.tgz", + "integrity": "sha512-CY6crGq313MX8GkwvB7tzgp99vjQxY1++5y10/BKN/GUfHqWaOGQMNZkBvqSzsZKWk/ijwHlWzzkLulsGHhjWQ==", "funding": [ { "type": "github", diff --git a/util/toolkit-js/package-lock.json b/util/toolkit-js/package-lock.json index fb2ecb34a..d5e5ae874 100644 --- a/util/toolkit-js/package-lock.json +++ b/util/toolkit-js/package-lock.json @@ -2880,9 +2880,9 @@ "license": "MIT" }, "node_modules/nanoid": { - "version": "3.3.12", - "resolved": "https://registry.npmjs.org/nanoid/-/nanoid-3.3.12.tgz", - "integrity": "sha512-ZB9RH/39qpq5Vu6Y+NmUaFhQR6pp+M2Xt76XBnEwDaGcVAqhlvxrl3B2bKS5D3NH3QR76v3aSrKaF/Kiy7lEtQ==", + "version": "3.3.18", + "resolved": "https://registry.npmjs.org/nanoid/-/nanoid-3.3.18.tgz", + "integrity": "sha512-DTg4MJbGMWkfi6VZFdNt2/caMbQy4Ou+Op/hJQvGEWcnVfoA1QA+xzRKAzw9jD6+GVOOeYr/mIcuDSdug6F6+w==", "dev": true, "funding": [ { @@ -3026,9 +3026,9 @@ } }, "node_modules/postcss": { - "version": "8.5.15", - "resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.15.tgz", - "integrity": "sha512-FfR8sjd4em2T6fb3I2MwAJU7HWVMr9zba+enmQeeWFfCbm+UOC/0X4DS8XtpUTMwWMGbjKYP7xjfNekzyGmB3A==", + "version": "8.5.26", + "resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.26.tgz", + "integrity": "sha512-u82N74LFzG8ca+dD8puPnplTXoGH4fTPpVGuIbt36G3qvNlkvfD0lEAZSxaly3KX8TS/L1A1gsCEmvKmBcVbkQ==", "dev": true, "funding": [ { @@ -3046,7 +3046,7 @@ ], "license": "MIT", "dependencies": { - "nanoid": "^3.3.12", + "nanoid": "^3.3.17", "picocolors": "^1.1.1", "source-map-js": "^1.2.1" }, @@ -3505,9 +3505,9 @@ } }, "node_modules/undici": { - "version": "7.28.0", - "resolved": "https://registry.npmjs.org/undici/-/undici-7.28.0.tgz", - "integrity": "sha512-cRZYrTDwWznlnRiPjggAGxZXanty6M8RV1ff8Wm4LWXBp7/IG8v5DnOm74DtUBp9OONpK75YlPnIjQqX0dBDtA==", + "version": "7.29.0", + "resolved": "https://registry.npmjs.org/undici/-/undici-7.29.0.tgz", + "integrity": "sha512-IDxfleLmmbSskfWSUATiN1nfn2rDuvnMOqb5CWR92iIfojA0Ud+ulOAAEQ57LPr9rWmsreUyf5lwyao+7GNNVw==", "license": "MIT", "engines": { "node": ">=20.18.1" @@ -3809,34 +3809,6 @@ "engines": { "node": ">=6" } - }, - "v7": { - "name": "@midnight-ntwrk/node-toolkit-v7", - "version": "0.1.0", - "extraneous": true, - "license": "Apache-2.0", - "dependencies": { - "@effect/cli": "^0.73.2", - "@effect/platform-node": "^0.104.1", - "@midnight-ntwrk/compact-js": "2.4.3", - "@midnight-ntwrk/compact-js-command": "2.4.3", - "@midnight-ntwrk/compact-js-node": "2.4.3", - "effect": "^3.21.0" - } - }, - "v8": { - "name": "@midnight-ntwrk/node-toolkit-v8", - "version": "0.1.1", - "extraneous": true, - "license": "Apache-2.0", - "dependencies": { - "@effect/cli": "^0.74.0", - "@effect/platform-node": "^0.105.0", - "@midnight-ntwrk/compact-js": "2.5.1", - "@midnight-ntwrk/compact-js-command": "2.5.1", - "@midnight-ntwrk/compact-js-node": "2.5.1", - "effect": "^3.21.0" - } } } } From 6c9d80c39f155c3c166c9aab94f292feff05076d Mon Sep 17 00:00:00 2001 From: Oscar Bailey <79094698+ozgb@users.noreply.github.com> Date: Thu, 13 Aug 2026 11:35:14 +0100 Subject: [PATCH 10/13] revert: drop the cNIGHT dust re-apply migration from this backport Reverts a5310e468 ("feat: reset Dust state and re-apply cNight UTxOs during migration (#2012)") and its follow-up 5b6bdc1fc, backing the cNIGHT dust generation replay out of the ledger-hardfork backport. PR #2012 is still open on main, so it lands there rather than here. Note: `metadata/static/midnight_metadata*.scale` were regenerated in 2ff2b4406 while the reverted storage items (`PreForkStateKey`, `DustReapplyCtime`, `DustReapplyProgress`) were present, so metadata needs a rebuild. Assisted-by: Claude:claude-opus-5 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> --- .../added/cnight-dust-generation-reapply.md | 62 --- ledger/helpers/src/fork/fork_8_to_9.rs | 8 +- .../helpers/src/state_translation_v8_to_v9.rs | 29 +- .../src/versions/common/wallet/dust.rs | 47 --- ledger/src/host_api/dust_generation.rs | 234 ----------- ledger/src/host_api/ledger_9.rs | 31 -- ledger/src/host_api/mod.rs | 5 - pallets/cnight-observation/mock/src/mock.rs | 2 - .../mock/src/mock_with_capture.rs | 34 +- pallets/cnight-observation/src/lib.rs | 80 +--- pallets/cnight-observation/src/migrations.rs | 1 - .../cnight-observation/src/migrations/v2.rs | 364 ------------------ .../tests/dust_reapply_tests.rs | 338 ---------------- .../tests/migration_tests.rs | 101 +---- pallets/cnight-observation/tests/tests.rs | 43 +-- pallets/midnight/src/lib.rs | 6 +- primitives/midnight/src/lib.rs | 9 +- runtime/src/lib.rs | 13 +- util/toolkit/tests/hardfork_e2e.rs | 76 +--- 19 files changed, 37 insertions(+), 1446 deletions(-) delete mode 100644 changes/runtime/added/cnight-dust-generation-reapply.md delete mode 100644 ledger/src/host_api/dust_generation.rs delete mode 100644 pallets/cnight-observation/src/migrations/v2.rs delete mode 100644 pallets/cnight-observation/tests/dust_reapply_tests.rs diff --git a/changes/runtime/added/cnight-dust-generation-reapply.md b/changes/runtime/added/cnight-dust-generation-reapply.md deleted file mode 100644 index 0b16581ea..000000000 --- a/changes/runtime/added/cnight-dust-generation-reapply.md +++ /dev/null @@ -1,62 +0,0 @@ -#cnight #dust #migration -# Re-apply cNIGHT dust generation after the ledger 8 -> 9 hardfork - -The ledger 8 -> 9 hardfork wipes dust state, which would silently stop DUST -generation for every cNIGHT holder. `pallet-cnight-observation` now rebuilds its -own slice of the ledger's dust generating set as a multi-block migration -(storage version 1 -> 2): - -- a single-block migration run in the upgrade block saves the pre-fork ledger-8 - arena root, the only place the wiped entries' night value and dust owner - survive; -- the multi-block migration then pages through `UtxoOwners` (the set of nonces - that are cnight's and still live), reads each nonce's pre-wipe value and owner - through a new `dust_generation_values_v8` host function (which also serves the - state's dust `time_to_cap`), and re-applies them in batches of 200 as - `CNightGeneratesDustUpdate` system transactions; -- incoming Cardano observations are ignored while it runs — the existing storage - version gate in `process_tokens` covers it, so `NextCardanoPosition` does not - advance and the observer re-delivers the same UTXOs afterwards. - -It is paced at one batch per block on purpose. `process_tokens`' benchmark -observes registration UTXOs, which never reach the ledger, so its weight says -nothing about the cost of 200 dust `Create`s; left to the weight meter the MBM -service budget would service ~100 batches — 20k ledger creates — in one block. -Measured live sets on 2026-08-06: **mainnet 4870** nonces (finalized #2019697), -**preview 1524** (#301994), **preprod 85** (#1985972) — so ~25 blocks (~2.5 min) -of gated observation on mainnet, ~8 on preview, 1 on preprod. - -The restored generation entries are field-for-field identical to the wiped ones. -Only the accrual clock moves: the original creation time lives on the dust UTXO -the wipe takes, so the replay stamps `fork block time - dust.time_to_cap()` -(~1 week). DUST accrues linearly from the creation time to a cap of -`night_value * night_dust_ratio` reached after `time_to_cap`, so backdating by -exactly that much puts every holder at their cap the moment the replay lands — -the pre-fork steady state, since anyone who had held cNIGHT for a week was -already capped. Stamping the fork block itself would instead start everyone at -zero and refill over a week in proportion to holdings, locking small holders out -of paying fees for days. The real per-UTXO creation time is only available from -db-sync, and would restore holders to the same cap anyway for all but the -youngest UTXOs, at the price of a new consensus-critical mainchain query in the -middle of the hardfork; the chosen offset over-credits only cNIGHT locked in the -last week, bounded by a cap it would have reached regardless. - -**Only cnight's slice of the generating set is restored.** Native NIGHT registers -generation entries too (at dust registration, for delegated unshielded outputs, -and on the mint/claim path); nothing in this repo records which of those the wipe -took, so `DustReapplyCompleted` must not be read as "all dust generation -restored". - -The wipe itself is part of this change: the v8 -> v9 state translation now drops -the dust state and installs the empty one genesis starts from, instead of -carrying it across. The toolkit's `fork_context_8_to_9` mirrors that, resetting -every wallet's local dust state so it does not try to spend dust the chain no -longer has. Should a translation ever carry dust across again, the migration -self-cancels rather than corrupting state: the first replayed event collides with -`GenerationInfoAlreadyPresent` and it emits `DustReapplySkipped`. - -New events: `DustReapplyStarted`, `DustReapplyBatchFailed`, -`DustReapplyCompleted`, `DustReapplySkipped`. - -PR: https://github.com/midnightntwrk/midnight-node/pull/2019 -Upstream PR: https://github.com/midnightntwrk/midnight-node/pull/2012 diff --git a/ledger/helpers/src/fork/fork_8_to_9.rs b/ledger/helpers/src/fork/fork_8_to_9.rs index 540323ebc..fac03ec49 100644 --- a/ledger/helpers/src/fork/fork_8_to_9.rs +++ b/ledger/helpers/src/fork/fork_8_to_9.rs @@ -114,7 +114,7 @@ pub fn fork_context_8_to_9( }) }) .transpose(); - let mut new_wallet = Wallet { + let new_wallet = Wallet { root_seed: v.root_seed.as_ref().map(|s| { WalletSeed::try_from(s.as_bytes()) .expect("wallet seed different length between versions") @@ -135,12 +135,6 @@ pub fn fork_context_8_to_9( dust: (*old_to_new_sp::<_, DustWallet>(crate::ledger_8::Sp::new(v.dust.clone()))?) .clone(), }; - // The fork wipes the on-chain dust state (see - // `state_translation_v8_to_v9`), so the wallet's view of its dust — UTxOs, - // generation info, merkle witnesses — is stale the moment we cross it. - // Reset it to a freshly-derived state under the v9 dust parameters; dust - // re-accrues from the post-fork generation entries the chain replays. - new_wallet.dust.wipe_local_state(&ledger_state.parameters); let new_key: WalletSeed = old_to_new_ser_untagged(&k)?; wallets.insert(new_key, new_wallet); } diff --git a/ledger/helpers/src/state_translation_v8_to_v9.rs b/ledger/helpers/src/state_translation_v8_to_v9.rs index 7b9f9ac7e..4106b7819 100644 --- a/ledger/helpers/src/state_translation_v8_to_v9.rs +++ b/ledger/helpers/src/state_translation_v8_to_v9.rs @@ -32,10 +32,8 @@ //! | ContractOperation | `contract-operation[v4]` | `contract-operation[v6]` | single `v2` key -> `{ v2, v3, ir }`; v8 key maps to `v2`, new `v3`/`ir` empty | //! | ContractMaintenanceAuthority | `contract-maintenance-authority[v1]` | `contract-maintenance-authority[v2]` | `committee: Vec` -> `Vec` (Schnorr/ECDSA sum) | //! -//! Everything else (zswap, utxo, replay_protection, treasury, -//! unclaimed_block_rewards) is tag-stable and passes through `recast`. `dust` -//! is the exception: it is tag-stable but deliberately *wiped* rather than -//! carried over (see [`LedgerStateTl::finalize`]). +//! Everything else (zswap, utxo, dust, replay_protection, treasury, +//! unclaimed_block_rewards) is tag-stable and passes through `recast`. // Map the upstream translation crate names onto the node workspace's package // aliases. `mn-ledger-8`/`mn-ledger-9` are the two `midnight-ledger` majors; @@ -313,13 +311,7 @@ impl contract: Map { mpt: contract_mpt.force_downcast(), key_type: PhantomData }, utxo: recast(&source.utxo)?, replay_protection: recast(&source.replay_protection)?, - // The hardfork wipes dust: the v8 dust state is dropped and replaced - // with the same empty state genesis starts from. Dust generation for - // still-locked cNIGHT is re-applied afterwards by - // `pallet_cnight_observation::migrations::v2`; dust UTxOs (balances) - // are not restored — they regenerate from the re-applied generation - // entries. - dust: Sp::new(ledger_v9::dust::DustState::default()), + dust: recast(&source.dust)?, })) } } @@ -715,19 +707,4 @@ mod tests { serialize::tagged_deserialize(&mut &buf[..]).expect("v9 deserialize"); assert_eq!(v9_rt.network_id, v9.network_id); } - - /// The translation wipes dust: whatever generation/utxo state v8 held, the - /// v9 side comes out as the empty state genesis starts from. - #[test] - fn dust_state_is_wiped() { - let mut v8 = ledger_v8::structure::LedgerState::::new("test-network"); - let mut dust = (*v8.dust).clone(); - dust.generation.generating_tree_first_free = 7; - dust.utxo.commitments_first_free = 3; - v8.dust = Sp::new(dust); - - let v9 = translate_to_completion(v8); - - assert_eq!(*v9.dust, ledger_v9::dust::DustState::default()); - } } diff --git a/ledger/helpers/src/versions/common/wallet/dust.rs b/ledger/helpers/src/versions/common/wallet/dust.rs index 758727552..f05461afa 100644 --- a/ledger/helpers/src/versions/common/wallet/dust.rs +++ b/ledger/helpers/src/versions/common/wallet/dust.rs @@ -69,21 +69,6 @@ impl DustWallet { Ok(Self::from_seed(derived_seed, params)) } - /// Drop everything this wallet knows about its dust — UTxOs, generation - /// info, merkle witnesses, in-flight spends — keeping only its keys, as if - /// it had just been derived. Used to mirror a hardfork that wipes the - /// on-chain dust state (ledger 8 -> 9): keeping the old local state would - /// have the wallet spend dust the chain no longer has. - /// - /// A watch-only wallet (no local state) stays watch-only. - pub fn wipe_local_state(&mut self, params: &LedgerParameters) { - self.dust_local_state = self - .dust_local_state - .as_ref() - .map(|_| Sp::new(DustLocalState::new(params.dust))); - self.spent_utxos = HashSet::new(); - } - pub fn replay_events<'a>( &mut self, events: impl IntoIterator>, @@ -233,38 +218,6 @@ mod tests { let path = DerivationPath::default_for_role(Role::UnshieldedExternal); assert!(DustWallet::::from_path(test_seed(), &path, None).is_err()); } - - #[test] - fn wipe_local_state_resets_state_but_keeps_keys() { - let params = super::super::super::INITIAL_PARAMETERS; - let mut wallet = DustWallet::::default(test_seed(), Some(¶ms)); - let public_key = wallet.public_key; - - // Stand in for a synced-up wallet: a non-default local state, as it would - // look after replaying pre-fork dust events. - let mut synced = (**wallet.dust_local_state.as_ref().unwrap()).clone(); - synced.sync_time = super::Timestamp::from_secs(1_000); - wallet.dust_local_state = Some(super::Sp::new(synced)); - - wallet.wipe_local_state(¶ms); - - let wiped = wallet.dust_local_state.as_ref().unwrap(); - assert_eq!(wiped.sync_time, super::Timestamp::default()); - assert_eq!(wallet.public_key, public_key, "keys must survive the wipe"); - assert!( - wallet - .speculative_spend(1, super::Timestamp::from_secs(1_000), ¶ms.dust) - .is_ok() - ); - } - - /// A watch-only wallet (address-derived, no local state) must stay that way. - #[test] - fn wipe_local_state_leaves_watch_only_wallets_alone() { - let mut wallet = DustWallet::::default(test_seed(), None); - wallet.wipe_local_state(&super::super::super::INITIAL_PARAMETERS); - assert!(wallet.dust_local_state.is_none()); - } } #[derive(Debug, Error)] diff --git a/ledger/src/host_api/dust_generation.rs b/ledger/src/host_api/dust_generation.rs deleted file mode 100644 index 8a367c64f..000000000 --- a/ledger/src/host_api/dust_generation.rs +++ /dev/null @@ -1,234 +0,0 @@ -// This file is part of midnight-node. -// Copyright (C) 2025-2026 Midnight Foundation -// SPDX-License-Identifier: Apache-2.0 -// Licensed under the Apache License, Version 2.0 (the "License"); -// You may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -//! Host-side batched read of the *pre-fork* (ledger-8) dust generation state. -//! -//! The ledger 8 -> 9 hardfork wipes dust state, so `pallet-cnight-observation` -//! has to re-apply the cNIGHT-generates-DUST entries it fed the ledger over the -//! chain's life (see `pallet_cnight_observation::migrations::v2`). Its own -//! storage records only which nonces are cnight's (`UtxoOwners`) — the night -//! `value` and the dust `owner` of each entry live in the about-to-be-wiped -//! `DustGenerationInfo`, which is what this module reads back out. -//! -//! It reads the *v8* state through the arena root the migration saved before -//! translation (`pallet_cnight_observation::PreForkStateKey`), which stays -//! resolvable because the arena retains historical ledger states. Reading a v8 -//! state from the v9 bridge is the same trick `serve_pre_migration_v8_read!` -//! plays in [`crate::host_api::ledger_9`]. - -use crate::ledger_8::api::Ledger as Ledger8; -use crate::ledger_9::types::{DeserializationError, LedgerApiError}; - -use base_crypto::{hash::HashOutput, time::Timestamp}; -use ledger_storage_ledger_8 as storage; -use midnight_serialize::{Serializable, tagged_deserialize}; -use mn_ledger_8::dust::InitialNonce; -use storage::{ - arena::{Sp, TypedArenaKey}, - db::DB, - storage::default_storage, -}; - -const LOG_TARGET: &str = "midnight::ledger::dust_generation"; - -/// The dust parameters' `time_to_cap` in seconds, plus — for each requested -/// initial nonce — the night value and dust owner of its still-generating entry -/// in the ledger-8 dust state referenced by `state_key`; `None` when the nonce -/// is not tracked, or has already been destroyed. Positionally aligned with -/// `nonces`. -/// -/// `time_to_cap` is how far the caller backdates the replayed `ctime` so every -/// restored entry lands at its DUST cap, i.e. at the balance it held before the -/// wipe. It comes from the v8 state because that is the one already loaded here, -/// and the 8 -> 9 translation recasts `parameters.dust` unchanged. -/// -/// The owner bytes are the (untagged) serialized `DustPublicKey`, i.e. exactly -/// what `construct_cnight_generates_dust_event` accepts for `owner`. -/// -/// Errors with `NoLedgerState` when `state_key` is not a ledger-8 state. This -/// must *not* be an all-`None` success: the caller's key is only v8 because its -/// `RecordPreForkState` migration runs before the pallet-midnight translation, -/// so a future reorder of the runtime `Migrations` tuple would otherwise -/// silently restore nothing, chain-wide, detectable only by absent DUST. -pub fn dust_generation_values_v8( - state_key: &[u8], - nonces: &[[u8; 32]], -) -> Result<(u64, Vec)>>), LedgerApiError> { - if !crate::is_ledger_8_state_key(state_key) { - log::error!( - target: LOG_TARGET, - "pre-fork state key is not a ledger-8 arena root; refusing to serve dust generation values" - ); - return Err(LedgerApiError::NoLedgerState); - } - - let key8: TypedArenaKey, D::Hasher> = tagged_deserialize(&mut &state_key[..]) - .map_err(|e| { - log::error!(target: LOG_TARGET, "failed to deserialize v8 state key: {e:?}"); - LedgerApiError::Deserialization(DeserializationError::TypedArenaKey) - })?; - // One arena load, amortised over the whole batch. - let ledger8: Sp, D> = default_storage::().arena.get_lazy(&key8).map_err(|e| { - log::error!(target: LOG_TARGET, "failed to load v8 ledger from arena: {e:?}"); - LedgerApiError::NoLedgerState - })?; - let generation = &ledger8.state.dust.generation; - // Non-negative by construction (`night_dust_ratio / generation_decay_rate`). - let time_to_cap = ledger8.state.parameters.dust.time_to_cap().as_seconds().max(0) as u64; - - let values = nonces - .iter() - .map(|nonce| { - // Same lookup path the ledger's own `Destroy` handler takes: - // nonce -> leaf index -> generating tree leaf. - let Some(index) = generation.night_indices.get(&InitialNonce(HashOutput(*nonce))) else { - // The caller only asks about nonces it believes are live, so an - // untracked one is a pallet/ledger divergence, same as a destroyed - // one below. - log::warn!(target: LOG_TARGET, "nonce {} is not tracked in the v8 dust state", hex::encode(nonce)); - return None; - }; - let Some((_, info)) = generation.generating_tree.index(*index) else { - log::error!( - target: LOG_TARGET, - "invariant violated: `night_indices` entry for {} not backed in `generating_tree`", - hex::encode(nonce), - ); - return None; - }; - // A destroyed entry keeps its leaf forever, with `dtime` rewritten. - // The caller's `UtxoOwners` is meant to be exactly the live set, so - // this is a pallet/ledger divergence worth surfacing. - if info.dtime != Timestamp::MAX { - log::warn!( - target: LOG_TARGET, - "nonce {} is already destroyed in the v8 dust state (dtime {:?}); not restoring", - hex::encode(nonce), - info.dtime, - ); - return None; - } - let mut owner = Vec::new(); - if let Err(e) = Serializable::serialize(&info.owner, &mut owner) { - log::error!(target: LOG_TARGET, "failed to serialize dust owner: {e:?}"); - return None; - } - Some((info.value, owner)) - }) - .collect(); - - Ok((time_to_cap, values)) -} - -#[cfg(test)] -mod tests { - use super::*; - use crate::ledger_9::api::Ledger as Ledger9; - use ledger_storage_ledger_8::db::InMemoryDB; - use midnight_serialize::tagged_serialize; - use mn_ledger_8::{ - dust::DustPublicKey, - structure::{ - CNightGeneratesDustActionType, CNightGeneratesDustEvent, LedgerState, SystemTransaction, - }, - }; - use transient_crypto::curve::Fr; - - const NONCE_LIVE: [u8; 32] = [1; 32]; - const NONCE_DESTROYED: [u8; 32] = [2; 32]; - const NONCE_UNKNOWN: [u8; 32] = [3; 32]; - - fn event( - nonce: [u8; 32], - value: u128, - owner: DustPublicKey, - action: CNightGeneratesDustActionType, - time_secs: u64, - ) -> CNightGeneratesDustEvent { - CNightGeneratesDustEvent { - value, - owner, - time: Timestamp::from_secs(time_secs), - action, - nonce: InitialNonce(HashOutput(nonce)), - } - } - - /// Persist `ledger` into the default in-memory arena and return its root, in - /// the same `StateKey` shape the pallet stores. - fn root_of + midnight_serialize::Tagged>(value: T) -> Vec { - let mut sp = default_storage::().arena.alloc(value); - sp.persist(); - let mut bytes = Vec::new(); - tagged_serialize(&sp.as_typed_key(), &mut bytes).expect("serialize root"); - bytes - } - - /// A v8 dust state holding a live and a destroyed cnight entry. - fn v8_root(owner: DustPublicKey) -> Vec { - use CNightGeneratesDustActionType::*; - let state = LedgerState::::new("local-test"); - let (state, _) = state - .apply_system_tx( - &SystemTransaction::CNightGeneratesDustUpdate { - events: vec![ - event(NONCE_LIVE, 100, owner, Create, 1_000), - event(NONCE_DESTROYED, 200, owner, Create, 1_000), - event(NONCE_DESTROYED, 200, owner, Destroy, 2_000), - ], - }, - Timestamp::from_secs(2_000), - ) - .expect("apply v8 cnight dust update"); - root_of(Ledger8::new(state)) - } - - #[test] - fn serves_live_entries_and_skips_destroyed_and_unknown() { - let owner = DustPublicKey(Fr::from(7u64)); - let root = v8_root(owner); - - let (time_to_cap, values) = dust_generation_values_v8::( - &root, - &[NONCE_LIVE, NONCE_DESTROYED, NONCE_UNKNOWN], - ) - .expect("v8 root must resolve"); - - let mut expected_owner = Vec::new(); - Serializable::serialize(&owner, &mut expected_owner).unwrap(); - - assert_eq!(values, vec![Some((100u128, expected_owner)), None, None]); - assert_eq!( - time_to_cap, - mn_ledger_8::structure::INITIAL_PARAMETERS.dust.time_to_cap().as_seconds() as u64, - "the served cap offset must be the state's own dust parameter", - ); - } - - /// The migration only ever holds a v8 key by construction, so a v9 key means - /// the migration order changed underneath us — that must fail loudly rather - /// than restore nothing. - #[test] - fn v9_state_key_is_rejected() { - let root = root_of(Ledger9::new(mn_ledger_9::structure::LedgerState::::new( - "local-test", - ))); - - assert!(matches!( - dust_generation_values_v8::(&root, &[NONCE_LIVE]), - Err(LedgerApiError::NoLedgerState), - )); - } -} diff --git a/ledger/src/host_api/ledger_9.rs b/ledger/src/host_api/ledger_9.rs index e696fdacd..982082939 100644 --- a/ledger/src/host_api/ledger_9.rs +++ b/ledger/src/host_api/ledger_9.rs @@ -575,37 +575,6 @@ pub trait Ledger9Bridge { } } - /// The dust `time_to_cap` in seconds, plus the night value and dust owner of - /// each requested nonce's still-generating entry in the *pre-fork* - /// (ledger-8) dust state, positionally aligned with `nonces` and `None` for - /// nonces that are untracked or already destroyed. - /// - /// Called by `pallet-cnight-observation`'s dust re-apply migration, which - /// rebuilds the cNIGHT generation entries the ledger 8 -> 9 hardfork wipes - /// and backdates their `ctime` by `time_to_cap`. `state_key` is the v8 arena - /// root it saved during the upgrade block; `Err(NoLedgerState)` means that - /// root no longer resolves, or is not a ledger-8 root at all. - fn dust_generation_values_v8( - &mut self, - state_key: PassFatPointerAndRead<&[u8]>, - nonces: PassFatPointerAndDecode>, - ) -> AllocateAndReturnByCodec)>>), LedgerApiError>> { - // The migration runs in `inherents_applied()`, after pallet-midnight has - // initialized the arena, but `set_default_storage` is idempotent and - // keeps this callable from anywhere in the block. - if is_unified(*self) { - Bridge::::set_default_storage(*self); - crate::host_api::dust_generation::dust_generation_values_v8::( - state_key, &nonces, - ) - } else { - Bridge::::set_default_storage(*self); - crate::host_api::dust_generation::dust_generation_values_v8::( - state_key, &nonces, - ) - } - } - /// Initialize a process-wide temporary ledger ParityDb seeded with the /// undeployed-network genesis state. /// diff --git a/ledger/src/host_api/mod.rs b/ledger/src/host_api/mod.rs index 778f942a3..1b1fd44c3 100644 --- a/ledger/src/host_api/mod.rs +++ b/ledger/src/host_api/mod.rs @@ -18,8 +18,3 @@ pub mod ledger_9; /// Host-side v8 -> v9 ledger state translation used by the runtime storage migration. #[cfg(feature = "std")] pub mod migration_8_to_9; - -/// Host-side read of the pre-fork (ledger-8) dust generation state, used by the -/// cNIGHT dust re-apply migration. -#[cfg(feature = "std")] -pub mod dust_generation; diff --git a/pallets/cnight-observation/mock/src/mock.rs b/pallets/cnight-observation/mock/src/mock.rs index b063b0134..bd667521f 100644 --- a/pallets/cnight-observation/mock/src/mock.rs +++ b/pallets/cnight-observation/mock/src/mock.rs @@ -148,8 +148,6 @@ parameter_types! { impl pallet_cnight_observation::Config for Test { type MidnightSystemTransactionExecutor = MidnightSystem; - type LedgerStateProvider = Midnight; - type LedgerBlockContextProvider = Midnight; type WeightInfo = (); } diff --git a/pallets/cnight-observation/mock/src/mock_with_capture.rs b/pallets/cnight-observation/mock/src/mock_with_capture.rs index 1c8e3ac52..7d6cfa39b 100644 --- a/pallets/cnight-observation/mock/src/mock_with_capture.rs +++ b/pallets/cnight-observation/mock/src/mock_with_capture.rs @@ -20,10 +20,7 @@ use frame_support::sp_runtime::{ use frame_support::traits::{ConstU16, ConstU32, ConstU64}; use frame_support::weights::RuntimeDbWeight; use frame_support::*; -use midnight_node_ledger::latest::types::BlockContext; -use midnight_primitives::{ - LedgerBlockContextProvider, LedgerStateProvider, MidnightSystemTransactionExecutor, -}; +use midnight_primitives::MidnightSystemTransactionExecutor; use sidechain_domain::*; #[cfg(feature = "std")] use sp_io::TestExternalities; @@ -168,37 +165,8 @@ impl MidnightSystemTransactionExecutor for MidnightSystemTx { } } -parameter_types! { - /// Stand-ins for `pallet-midnight`, which this mock deliberately omits (it - /// exists to capture system transactions without a ledger). Tests drive the - /// dust replay migration through these. - pub static MockLedgerStateKey: Vec = Vec::new(); - pub static MockBlockTime: u64 = 1_700_000_000; -} - -pub struct MockLedger; - -impl LedgerStateProvider for MockLedger { - fn get_ledger_state_key() -> Vec { - MockLedgerStateKey::get() - } -} - -impl LedgerBlockContextProvider for MockLedger { - fn get_block_context() -> BlockContext { - BlockContext { - tblock: MockBlockTime::get(), - tblock_err: 0, - parent_block_hash: vec![0u8; 32], - last_block_time: 0, - } - } -} - impl pallet_cnight_observation::Config for Test { type MidnightSystemTransactionExecutor = MidnightSystemTx; - type LedgerStateProvider = MockLedger; - type LedgerBlockContextProvider = MockLedger; type WeightInfo = (); } diff --git a/pallets/cnight-observation/src/lib.rs b/pallets/cnight-observation/src/lib.rs index 91788cd87..7e40033f8 100644 --- a/pallets/cnight-observation/src/lib.rs +++ b/pallets/cnight-observation/src/lib.rs @@ -84,9 +84,7 @@ pub const MAX_UTXO_COUNT: u32 = DEFAULT_CARDANO_TX_CAPACITY_PER_BLOCK * UTXO_PER #[frame_support::pallet] pub mod pallet { use frame_support::sp_runtime::traits::Hash; - use midnight_primitives::{ - LedgerBlockContextProvider, LedgerStateProvider, MidnightSystemTransactionExecutor, - }; + use midnight_primitives::MidnightSystemTransactionExecutor; use midnight_primitives_cnight_observation::{ CARDANO_ASSET_NAME_MAX_LENGTH, CARDANO_BECH32_ADDRESS_MAX_LENGTH, CNIGHT_POLICY_ID_LENGTH, CardanoRewardAddressBytes, DustPublicKeyBytes, @@ -151,9 +149,7 @@ pub mod pallet { pub system_transaction_hash: LedgerHash, } - // v2: re-apply the cNIGHT dust generation entries the ledger 8 -> 9 hardfork - // wipes (see `migrations::v2`). - const STORAGE_VERSION: StorageVersion = StorageVersion::new(2); + const STORAGE_VERSION: StorageVersion = StorageVersion::new(1); #[pallet::pallet] #[pallet::storage_version(STORAGE_VERSION)] @@ -163,11 +159,6 @@ pub mod pallet { #[pallet::config] pub trait Config: frame_system::Config { type MidnightSystemTransactionExecutor: MidnightSystemTransactionExecutor; - /// Reads the ledger state key, to capture the pre-hardfork (ledger-8) - /// one before the pallet-midnight translation replaces it. - type LedgerStateProvider: LedgerStateProvider; - /// Supplies the ledger time stamped on the replayed dust events. - type LedgerBlockContextProvider: LedgerBlockContextProvider; /// Weight information for extrinsics in this pallet. type WeightInfo: crate::weights::WeightInfo; } @@ -180,27 +171,6 @@ pub mod pallet { MappingAdded(MappingEntry), MappingRemoved(MappingEntry), SystemTransactionApplied(SystemTransactionApplied), - /// The hardfork upgrade block armed the dust generation replay - /// (`migrations::v2`) by saving the pre-fork ledger state key. - DustReapplyStarted, - /// One replay batch failed to apply; its nonces were not restored. The - /// replay continues with the next batch. - DustReapplyBatchFailed { - nonces: Vec, - }, - /// The replay finished. `applied` entries were restored; `skipped` were - /// not (untracked, already destroyed, or in a failed batch). - /// - /// Note this covers cnight's slice of the ledger's dust generating set - /// only — native-NIGHT generation entries are not restored here. - DustReapplyCompleted { - applied: u32, - skipped: u32, - }, - /// The replay did not run: the hardfork did not wipe dust state, no - /// pre-fork state key was recorded, or that key is unreadable. The - /// reason is logged. - DustReapplySkipped, } #[pallet::error] @@ -324,25 +294,6 @@ pub mod pallet { #[pallet::storage] pub type InherentExecutedThisBlock = StorageValue<_, bool, ValueQuery>; - /// The ledger-8 arena root as of the hardfork upgrade block, retained so the - /// dust replay (`migrations::v2`) can read pre-wipe night values and owners - /// after `pallet_midnight::StateKey` has moved on to the v9 root. Mirrors - /// that item's shape. Killed when the replay finishes. - #[pallet::storage] - #[pallet::unbounded] - pub type PreForkStateKey = StorageValue<_, Vec, OptionQuery>; - - /// Ledger time stamped on every replayed dust event: the fork block's own - /// time, backdated by the dust `time_to_cap` so every restored entry lands - /// at its DUST cap rather than at zero. Written by the first replay step. - #[pallet::storage] - pub type DustReapplyCtime = StorageValue<_, u64, OptionQuery>; - - /// Running (applied, skipped) tallies of the dust replay — the only on-chain - /// evidence it ran to completion. Killed when the replay finishes. - #[pallet::storage] - pub type DustReapplyProgress = StorageValue<_, (u32, u32), ValueQuery>; - #[pallet::genesis_config] #[derive(frame_support::DefaultNoBound)] pub struct GenesisConfig { @@ -706,24 +657,15 @@ pub mod pallet { ensure!(!InherentExecutedThisBlock::::get(), Error::::InherentAlreadyExecuted); InherentExecutedThisBlock::::put(true); - // Skip observation processing entirely while any multi-block migration of - // this pallet's storage is in flight; `NextCardanoPosition` stays - // unchanged so the next block's inherent re-presents the same UTXOs (plus - // any new ones) and we resume once the migration finishes. - // - // v0 -> v1 (`Mapping`): `unique_dust_key` (and therefore - // `handle_registration`, `handle_registration_removal`, `handle_create`) - // reads only v1, missing any v0 row not yet moved. Acting on that partial - // view would silently corrupt registration state — e.g. a deregistration - // whose v0 row is still pending would no-op here and then re-appear as - // live once the migration drains it. - // - // v1 -> v2 (dust generation replay): the replay re-applies `Create` - // events for every live `UtxoOwners` nonce. A concurrent spend would - // `take` a nonce the replay has not restored yet (its `Destroy` failing - // against a wiped ledger, then the nonce gone from the live set), and a - // concurrent create would race the replay's own system transaction. - + // While a multi-block migration of `Mapping` is still draining v0 storage, + // `unique_dust_key` (and therefore `handle_registration`, + // `handle_registration_removal`, `handle_create`) reads only v1, missing + // any v0 row that hasn't been moved yet. Acting on that partial view would + // silently corrupt registration state — e.g. a deregistration whose v0 + // row is still pending would no-op here and then re-appear as live once + // the migration drains it. Skip processing entirely; `NextCardanoPosition` + // stays unchanged so the next block's inherent re-presents the same UTXOs + // (plus any new ones) and we resume once the migration finishes. if Pallet::::on_chain_storage_version() < STORAGE_VERSION { log::warn!( "cnight-observation: skipping process_tokens (on-chain storage version {:?} < {:?}); MBM in progress", diff --git a/pallets/cnight-observation/src/migrations.rs b/pallets/cnight-observation/src/migrations.rs index 5d8546f22..5ae2e0980 100644 --- a/pallets/cnight-observation/src/migrations.rs +++ b/pallets/cnight-observation/src/migrations.rs @@ -13,6 +13,5 @@ // limitations under the License. pub mod v1; -pub mod v2; pub const PALLET_MIGRATIONS_ID: &[u8; 25] = b"pallet-cnight-observation"; diff --git a/pallets/cnight-observation/src/migrations/v2.rs b/pallets/cnight-observation/src/migrations/v2.rs deleted file mode 100644 index 023c2523e..000000000 --- a/pallets/cnight-observation/src/migrations/v2.rs +++ /dev/null @@ -1,364 +0,0 @@ -// Copyright (C) Midnight Foundation -// SPDX-License-Identifier: Apache-2.0 -// Licensed under the Apache License, Version 2.0 (the "License"); -// You may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -//! Storage migration v1 → v2: re-apply cNIGHT dust generation after the -//! ledger 8 → 9 hardfork wipes dust state. -//! -//! Every cNIGHT UTXO this pallet observed fed a `Create` event into the ledger's -//! dust generating set. The hardfork wipes that state, so without this migration -//! every cNIGHT holder would silently stop generating DUST. Two parts: -//! -//! * [`RecordPreForkState`] — single-block, and **must run before** -//! `pallet_midnight::migrations::v2` in the runtime's `Migrations` tuple: it -//! saves the still-untranslated ledger-8 arena root, which is the only place -//! the wiped entries' night `value` and dust `owner` survive. -//! * [`MigrateV1ToV2`] — the multi-block replay. It pages through `UtxoOwners` -//! (read-only: the provenance-and-liveness filter for which nonces are -//! cnight's and still live), asks the host for each nonce's pre-wipe -//! `(value, owner)`, and applies one `CNightGeneratesDustUpdate` per step. -//! `process_tokens` is gated off for the duration by the storage version. -//! -//! The restored generation entries are field-for-field identical to the wiped -//! ones. Only the accrual clock moves: the original `ctime` is not publicly -//! visible in ledger state (it is stored as a commitment only), so the replay -//! stamps `fork block time - dust.time_to_cap()`. DUST accrues linearly from -//! `ctime` to a cap of `night_value * night_dust_ratio` reached after -//! `time_to_cap` (~1 week), so backdating by exactly that much puts every holder -//! at their cap the moment the replay lands — the pre-fork steady state, since -//! anyone holding cNIGHT for a week was already capped. -//! -//! Stamping the fork block itself instead would be an equally arbitrary clock -//! that starts everyone at zero and refills over a week, in proportion to -//! holdings: large holders recover in minutes, small ones are locked out of -//! paying fees for days. The real per-UTXO `ctime` is only available from -//! db-sync, and would restore holders to the same cap anyway for all but the -//! youngest UTXOs — while making the hardfork depend on a new -//! consensus-critical mainchain query. The chosen offset over-credits only -//! cNIGHT locked in the last week, bounded by a cap it would reach regardless. -//! -//! Only cnight's slice of the generating set is restored. Native NIGHT registers -//! generation entries too, and nothing in this repo records which of those the -//! wipe took. -//! -//! The wipe itself lives in the translation table -//! (`midnight_node_ledger_helpers::state_translation_v8_to_v9`), which replaces -//! the v8 dust state with the empty one. Should that ever stop being true, this -//! migration self-cancels rather than corrupting state: the first replayed -//! `Create` collides with `GenerationInfoAlreadyPresent` (see -//! [`MigrateV1ToV2::step`]). - -extern crate alloc; - -use alloc::vec::Vec; -use frame_support::{ - migrations::{MigrationId, SteppedMigration, SteppedMigrationError}, - pallet_prelude::*, - traits::OnRuntimeUpgrade, - weights::WeightMeter, -}; -use midnight_node_ledger::types::active_ledger_bridge as LedgerApi; -use midnight_primitives::{ - LedgerBlockContextProvider, LedgerStateProvider, MidnightSystemTransactionExecutor, -}; - -use super::PALLET_MIGRATIONS_ID; -use crate::{ - Config, DustReapplyCtime, DustReapplyProgress, Event, Pallet, PreForkStateKey, UtxoActionType, - UtxoOwners, -}; - -const LOG_TARGET: &str = "cnight-observation::migration"; - -/// Nonces restored per step, and hence per host call and per system transaction. -/// -/// Matches `DEFAULT_CARDANO_TX_CAPACITY_PER_BLOCK`: a batch this size is one -/// `CNightGeneratesDustUpdate` that `process_tokens` already applies in a single -/// block in production, so it is known to fit. It also bounds the blast radius -/// of a failed batch. -pub const MAX_REAPPLY_BATCH: u32 = 200; - -/// Saves the pre-hardfork ledger-8 arena root for [`MigrateV1ToV2`] to read the -/// wiped dust entries' values and owners from. -/// -/// Single-block and O(1). Must sit *before* `pallet_midnight::migrations::v2` in -/// the runtime `Migrations` tuple — that migration replaces -/// `pallet_midnight::StateKey` with the translated v9 root. -pub struct RecordPreForkState(core::marker::PhantomData); - -impl OnRuntimeUpgrade for RecordPreForkState { - fn on_runtime_upgrade() -> Weight { - let weight = T::DbWeight::get().reads_writes(2, 1); - - if Pallet::::on_chain_storage_version() >= 2 { - return weight; - } - if PreForkStateKey::::exists() { - // Should be impossible: `pallet_migrations` blocks `set_code` while - // an MBM is in flight, so the replay cannot still be holding a key. - log::error!( - target: LOG_TARGET, - "pre-fork ledger state key is already set; leaving it alone rather than overwriting" - ); - return weight; - } - - PreForkStateKey::::put(T::LedgerStateProvider::get_ledger_state_key()); - Pallet::::deposit_event(Event::::DustReapplyStarted); - log::info!(target: LOG_TARGET, "recorded pre-fork ledger state key for the dust generation replay"); - - weight - } -} - -/// Replays cnight's dust generation entries into the post-hardfork ledger state, -/// one `UtxoOwners` page per step. -pub struct MigrateV1ToV2(core::marker::PhantomData); - -impl SteppedMigration for MigrateV1ToV2 { - /// The last `UtxoOwners` nonce processed. - type Cursor = T::Hash; - type Identifier = MigrationId<25>; - - fn id() -> Self::Identifier { - MigrationId { pallet_id: *PALLET_MIGRATIONS_ID, version_from: 1, version_to: 2 } - } - - fn step( - cursor: Option, - meter: &mut WeightMeter, - ) -> Result, SteppedMigrationError> { - // One batch per step, and — by charging half a block — one step per block. - // - // The weight model cannot pace this: `process_tokens`' benchmark observes - // *registration* UTXOs, which never reach the ledger, so its ~15ms for 200 - // UTXOs says nothing about 200 dust `Create`s. Against - // `MbmServiceWeight` (80% of the block) that would service ~100 batches — - // 20k ledger dust creates — in a single block. Half a block is over the - // service budget for a second step and under it for the first, so exactly - // one batch lands per block, and never the fatal - // `required > MaxServiceWeight`. - // - // The cost is latency, and it is small: mainnet's live set was ~4.9k - // nonces on 2026-08-06 (preview ~1.5k, preprod ~85), i.e. ~25 batches, - // so ~25 blocks (~2.5 min) of gated observation. The observer re-delivers - // everything afterwards. - let required = Weight::from_parts(T::BlockWeights::get().max_block.ref_time() / 2, 0); - if meter.remaining().any_lt(required) { - return Err(SteppedMigrationError::InsufficientWeight { required }); - } - let _ = meter.try_consume(required); - - // Never return `Err` below this point: steps run under - // `FreezeChainOnFailedMigration`, so any error freezes the chain. Every - // failure path instead winds the replay up and lets the observer resume. - let Some(pre_fork_key) = PreForkStateKey::::get() else { - log::info!( - target: LOG_TARGET, - "no pre-fork ledger state key recorded; nothing to replay" - ); - return Ok(cancel::()); - }; - - // Read-only paging: `UtxoOwners` is not drained, it stays the live set. - let mut iter = match cursor { - Some(last) => UtxoOwners::::iter_from(UtxoOwners::::hashed_key_for(last)), - None => UtxoOwners::::iter(), - }; - let nonces: Vec = - iter.by_ref().take(MAX_REAPPLY_BATCH as usize).map(|(nonce, _)| nonce).collect(); - - let Some(last) = nonces.last().copied() else { - return Ok(complete::()); - }; - - let raw_nonces: Vec<[u8; 32]> = nonces.iter().map(|nonce| nonce.0).collect(); - let (time_to_cap, values) = - match LedgerApi::dust_generation_values_v8(&pre_fork_key, raw_nonces) { - Ok(values) => values, - Err(e) => { - // The pre-fork arena root has been reaped, or (defensively) is - // not a ledger-8 root at all. Nothing to restore from. - log::error!( - target: LOG_TARGET, - "pre-fork dust generation state is unreadable ({e:?}); abandoning the replay" - ); - return Ok(cancel::()); - }, - }; - - // Stamped once, on the first step that has something to restore, and - // reused by every later batch so the whole set shares one clock. Steps - // run in `inherents_applied()`, i.e. after the timestamp inherent, so - // `tblock` is the current block's own time; backdating it by - // `time_to_cap` puts every restored entry straight at its DUST cap. - let ctime = match DustReapplyCtime::::get() { - Some(ctime) => ctime, - None => { - let tblock = T::LedgerBlockContextProvider::get_block_context().tblock; - let ctime = tblock.saturating_sub(time_to_cap); - DustReapplyCtime::::put(ctime); - ctime - }, - }; - - let mut skipped = 0u32; - let mut events = Vec::with_capacity(nonces.len()); - for (nonce, value) in nonces.iter().zip(values) { - // `None`: the nonce is untracked in the v8 dust state, or was - // already destroyed there (both logged host-side). - let Some((night_value, owner)) = value else { - skipped = skipped.saturating_add(1); - continue; - }; - match LedgerApi::construct_cnight_generates_dust_event( - night_value, - &owner, - ctime, - UtxoActionType::Create as u8, - nonce.0, - ) { - Ok(event) => events.push(event), - Err(e) => { - log::error!(target: LOG_TARGET, "failed to construct replay event: {e:?}"); - skipped = skipped.saturating_add(1); - }, - } - } - - let (restored_so_far, _) = DustReapplyProgress::::get(); - let mut applied = events.len() as u32; - if !events.is_empty() && !apply_batch::(events) { - if restored_so_far == 0 { - // Nothing has been restored yet, so the likely reason is that the - // hardfork did not wipe dust after all: re-applying a surviving - // `Create` fails with `GenerationInfoAlreadyPresent`. This is the - // self-cancel that keeps the migration inert against a - // translation that carries dust across. (Keyed on "nothing - // restored" rather than "first - // batch" because a leading page can legitimately resolve to no - // events at all, and then never apply anything.) - log::warn!( - target: LOG_TARGET, - "replay batch failed with nothing restored yet; assuming dust state survived the hardfork and cancelling the replay" - ); - return Ok(cancel::()); - } - // A failed batch left the ledger state untouched (the ledger - // propagates the first event's error out of the whole system - // transaction, and `mut_ledger_state` only writes on success), so - // carrying on with the next page is safe. - Pallet::::deposit_event(Event::::DustReapplyBatchFailed { nonces }); - skipped = skipped.saturating_add(applied); - applied = 0; - } - - DustReapplyProgress::::mutate(|(total_applied, total_skipped)| { - *total_applied = total_applied.saturating_add(applied); - *total_skipped = total_skipped.saturating_add(skipped); - }); - - Ok(Some(last)) - } - - #[cfg(feature = "try-runtime")] - fn pre_upgrade() -> Result, sp_runtime::TryRuntimeError> { - // Count only: `UtxoOwners` is chain-scale, never snapshot it. - Ok((UtxoOwners::::iter_keys().count() as u64).encode()) - } - - #[cfg(feature = "try-runtime")] - fn post_upgrade(state: Vec) -> Result<(), sp_runtime::TryRuntimeError> { - use frame_support::ensure; - - let live: u64 = - Decode::decode(&mut state.as_slice()).expect("pre_upgrade count must decode"); - - ensure!( - Pallet::::on_chain_storage_version() == 2, - "storage version must be 2 after the dust replay" - ); - ensure!( - UtxoOwners::::iter_keys().count() as u64 == live, - "the dust replay must not change the live UtxoOwners set" - ); - ensure!( - PreForkStateKey::::get().is_none(), - "pre-fork ledger state key must be cleared after the dust replay" - ); - ensure!( - DustReapplyCtime::::get().is_none(), - "replay ctime must be cleared after the dust replay" - ); - ensure!( - DustReapplyProgress::::get() == (0, 0), - "replay progress must be cleared after the dust replay" - ); - - Ok(()) - } -} - -/// Applies one batch as a single `CNightGeneratesDustUpdate`, the same pair of -/// calls `process_tokens` makes. Returns false (having logged) on failure. -/// -/// `execute_system_transaction` deposits `pallet_midnight_system`'s own -/// `SystemTransactionApplied` event carrying the serialized transaction, which -/// is the indexer's hook — this pallet's variant is deliberately not emitted, -/// its `CmstHeader` being a Cardano position that has no meaning here. -fn apply_batch(events: Vec>) -> bool { - let tx = match LedgerApi::construct_cnight_generates_dust_system_tx(events) { - Ok(tx) => tx, - Err(e) => { - log::error!(target: LOG_TARGET, "failed to construct replay system tx: {e:?}"); - return false; - }, - }; - - match T::MidnightSystemTransactionExecutor::execute_system_transaction(tx) { - Ok(_) => true, - Err(e) => { - log::error!(target: LOG_TARGET, "replay batch failed to apply: {e:?}"); - false - }, - } -} - -/// Wind the replay up without restoring anything, and let the observer resume. -fn cancel() -> Option { - clear_transient::(); - Pallet::::deposit_event(Event::::DustReapplySkipped); - finish::() -} - -/// Wind the replay up after the last page, reporting the tallies. -fn complete() -> Option { - let (applied, skipped) = DustReapplyProgress::::get(); - clear_transient::(); - Pallet::::deposit_event(Event::::DustReapplyCompleted { applied, skipped }); - log::info!(target: LOG_TARGET, "dust generation replay complete: {applied} applied, {skipped} skipped"); - finish::() -} - -fn clear_transient() { - PreForkStateKey::::kill(); - DustReapplyCtime::::kill(); - DustReapplyProgress::::kill(); -} - -/// MBMs don't bump the pallet's `StorageVersion`; do it ourselves so -/// `process_tokens` starts accepting observations again. -fn finish() -> Option { - StorageVersion::new(2).put::>(); - None -} diff --git a/pallets/cnight-observation/tests/dust_reapply_tests.rs b/pallets/cnight-observation/tests/dust_reapply_tests.rs deleted file mode 100644 index aafb45887..000000000 --- a/pallets/cnight-observation/tests/dust_reapply_tests.rs +++ /dev/null @@ -1,338 +0,0 @@ -// This file is part of midnight-node. -// Copyright (C) Midnight Foundation -// SPDX-License-Identifier: Apache-2.0 -// Licensed under the Apache License, Version 2.0 (the "License"); -// You may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// http://www.apache.org/licenses/LICENSE-2.0 -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -//! v1 -> v2 dust generation replay tests, against a **real** ledger. -//! -//! The mock wires the real `pallet-midnight`/`pallet-midnight-system` over a -//! parity-db arena, so the replay's system transactions genuinely apply to a -//! ledger-9 state, and the pre-fork values genuinely come out of a ledger-8 one -//! seeded in the same arena (v8 and v9 share the storage backend). - -use frame_support::{migrations::SteppedMigration, pallet_prelude::*, weights::WeightMeter}; -use midnight_node_ledger_helpers::{ - CNightGeneratesDustActionType, DustPublicKey, SystemTransaction, deserialize, - serialize_untagged, -}; -use midnight_node_res::networks::{MidnightNetwork, UndeployedNetwork}; -use midnight_primitives_cnight_observation::DustPublicKeyBytes; -use pallet_cnight_observation::{ - DustReapplyCtime, DustReapplyProgress, Event, Pallet, PreForkStateKey, UtxoOwners, - migrations::v2::{MAX_REAPPLY_BATCH, MigrateV1ToV2}, -}; -use pallet_cnight_observation_mock::mock::{ - self, CNightObservation, RuntimeEvent, System, Test, new_test_ext, -}; -use sp_core::H256; -use test_log::test; - -/// v8-side ledger types, reached through the helpers crate's per-generation -/// re-exports (the same crates the node's ledger-8 module is built from). -mod v8 { - pub use midnight_node_ledger_helpers::ledger_8::{ - base_crypto::{hash::HashOutput, time::Timestamp}, - ledger_storage::{db::ParityDb, storage::default_storage}, - midnight_serialize::tagged_serialize, - mn_ledger::{ - dust::{DustPublicKey, InitialNonce}, - structure::{ - CNightGeneratesDustActionType, CNightGeneratesDustEvent, LedgerState, - SystemTransaction, - }, - }, - transient_crypto::curve::Fr, - }; -} - -/// The fork block's time. -const FORK_TIME_SECS: u64 = 1_800_000_000; - -/// The `ctime` every replayed entry must carry: the fork block backdated by the -/// dust `time_to_cap`, so each restored entry is at its DUST cap on arrival. -/// Derived from the *active* (v9) parameters — the 8 -> 9 translation recasts -/// `parameters.dust` unchanged, so this is an independent check of the value the -/// migration reads out of the v8 state. -fn expected_ctime_secs() -> u64 { - FORK_TIME_SECS - - midnight_node_ledger_helpers::INITIAL_PARAMETERS.dust.time_to_cap().as_seconds() as u64 -} - -fn init_ledger_state() { - let path_buf = tempfile::tempdir().unwrap().keep(); - let state_key = midnight_node_ledger::latest::storage::init_storage_paritydb_separate( - &path_buf, - UndeployedNetwork.genesis_state(), - 1024 * 1024, - ); - - mock::Midnight::initialize_state(UndeployedNetwork.id(), &state_key); - mock::System::set_block_number(1); - mock::Timestamp::set_timestamp(FORK_TIME_SECS * 1000); - StorageVersion::new(1).put::(); -} - -fn nonce(byte: u8) -> H256 { - H256([byte; 32]) -} - -fn owner() -> v8::DustPublicKey { - v8::DustPublicKey(v8::Fr::from(7u64)) -} - -fn owner_bytes() -> DustPublicKeyBytes { - DustPublicKeyBytes(serialize_untagged(&owner()).unwrap().try_into().unwrap()) -} - -/// Build a ledger-8 state whose dust generating set holds `entries`, persist it -/// into the (shared) arena and return its root — exactly what -/// `RecordPreForkState` would have saved during the hardfork upgrade block. -fn seed_pre_fork_state(entries: &[(H256, u128)]) -> Vec { - let events = entries - .iter() - .map(|(nonce, value)| v8::CNightGeneratesDustEvent { - value: *value, - owner: owner(), - time: v8::Timestamp::from_secs(FORK_TIME_SECS - 1_000), - action: v8::CNightGeneratesDustActionType::Create, - nonce: v8::InitialNonce(v8::HashOutput(nonce.0)), - }) - .collect(); - - let (state, _) = v8::LedgerState::::new(UndeployedNetwork.id()) - .apply_system_tx( - &v8::SystemTransaction::CNightGeneratesDustUpdate { events }, - v8::Timestamp::from_secs(FORK_TIME_SECS - 1_000), - ) - .expect("seed v8 dust generation entries"); - - let mut sp = v8::default_storage::() - .arena - .alloc(midnight_node_ledger::ledger_8::api::ledger::Ledger::new(state)); - sp.persist(); - let mut root = Vec::new(); - v8::tagged_serialize(&sp.as_typed_key(), &mut root).expect("serialize v8 root"); - root -} - -/// Drive the replay to completion, returning the number of steps taken. -fn run_to_completion() -> u32 { - let mut cursor = None; - let mut steps = 0; - loop { - let mut meter = WeightMeter::new(); - cursor = MigrateV1ToV2::::step(cursor, &mut meter).expect("step must not fail"); - steps += 1; - if cursor.is_none() { - return steps; - } - } -} - -/// The `CNightGeneratesDustEvent`s of every system transaction applied so far. -fn applied_dust_events() -> Vec { - System::events() - .iter() - .filter_map(|record| match &record.event { - RuntimeEvent::MidnightSystem( - pallet_midnight_system::Event::SystemTransactionApplied(applied), - ) => Some(applied.serialized_system_transaction.clone()), - _ => None, - }) - .flat_map(|tx| { - let SystemTransaction::CNightGeneratesDustUpdate { events } = - deserialize(&tx[..]).expect("deserialize replay system tx") - else { - panic!("replay must apply a CNightGeneratesDustUpdate"); - }; - events - }) - .collect() -} - -fn cnight_events() -> Vec> { - System::events() - .iter() - .filter_map(|record| match &record.event { - RuntimeEvent::CNightObservation(e) => Some(e.clone()), - _ => None, - }) - .collect() -} - -/// The happy path: every live `UtxoOwners` nonce is restored with the night value -/// and dust owner the pre-fork ledger held for it, stamped with a `ctime` that -/// puts it at its DUST cap. A nonce the pre-fork state doesn't know is tallied as -/// skipped. -#[test] -fn replays_live_entries_from_pre_fork_state() { - new_test_ext().execute_with(|| { - init_ledger_state(); - - let entries = [(nonce(1), 100u128), (nonce(2), 250u128), (nonce(3), 7u128)]; - PreForkStateKey::::put(seed_pre_fork_state(&entries)); - for (nonce, _) in entries.iter() { - UtxoOwners::::insert(nonce, owner_bytes()); - } - // Live in the pallet but absent from the pre-fork ledger state. - UtxoOwners::::insert(nonce(9), owner_bytes()); - - assert_eq!(run_to_completion(), 2, "one batch, then the completing step"); - - assert_eq!(cnight_events(), vec![Event::DustReapplyCompleted { applied: 3, skipped: 1 }],); - assert_eq!(Pallet::::on_chain_storage_version(), 2); - assert!(PreForkStateKey::::get().is_none()); - assert!(DustReapplyCtime::::get().is_none()); - assert_eq!(DustReapplyProgress::::get(), (0, 0)); - assert_eq!( - UtxoOwners::::iter().count(), - 4, - "UtxoOwners is the live set, not a queue — it must survive the replay", - ); - - // The applied events must be field-for-field the wiped ones, bar `ctime`. - let expected_owner: DustPublicKey = - midnight_node_ledger_helpers::deserialize_untagged(&mut &owner_bytes().0[..]).unwrap(); - let mut applied: Vec<(u128, [u8; 32])> = applied_dust_events() - .iter() - .map(|event| { - assert_eq!(event.action, CNightGeneratesDustActionType::Create); - assert_eq!(event.owner, expected_owner, "restored owner must be the ledger's"); - assert_eq!( - event.time.to_secs(), - expected_ctime_secs(), - "replayed ctime must be the fork block backdated by time_to_cap", - ); - (event.value, event.nonce.0.0) - }) - .collect(); - applied.sort(); - let mut expected: Vec<(u128, [u8; 32])> = - entries.iter().map(|(nonce, value)| (*value, nonce.0)).collect(); - expected.sort(); - assert_eq!(applied, expected); - }); -} - -/// The inert-today path, for real: re-applying entries that are still present -/// fails with `GenerationInfoAlreadyPresent` on the first batch, which is how the -/// replay detects that the hardfork did not wipe dust after all. Driven by -/// replaying twice — the second run's ledger state already holds the entries. -#[test] -fn first_batch_failure_self_cancels() { - new_test_ext().execute_with(|| { - init_ledger_state(); - - let entries = [(nonce(1), 100u128), (nonce(2), 250u128)]; - let pre_fork_key = seed_pre_fork_state(&entries); - PreForkStateKey::::put(pre_fork_key.clone()); - for (nonce, _) in entries.iter() { - UtxoOwners::::insert(nonce, owner_bytes()); - } - run_to_completion(); - - // Now the current (v9) state holds them, as it would if the hardfork had - // carried dust across instead of wiping it. - frame_system::Pallet::::reset_events(); - StorageVersion::new(1).put::(); - PreForkStateKey::::put(pre_fork_key); - - assert_eq!(run_to_completion(), 1, "the failing first batch must end the replay"); - - assert_eq!(cnight_events(), vec![Event::DustReapplySkipped]); - assert_eq!(Pallet::::on_chain_storage_version(), 2); - assert!(PreForkStateKey::::get().is_none()); - assert!(applied_dust_events().is_empty(), "nothing must have been applied"); - }); -} - -/// More rows than one batch: the cursor hands off between steps and every row is -/// visited exactly once (the tallies sum to the row count). -#[test] -fn pages_across_steps_visiting_every_row_once() { - new_test_ext().execute_with(|| { - init_ledger_state(); - - // Only a couple of rows resolve against the pre-fork state; the rest are - // tallied as skipped. Keeps the seeded v8 state small while still - // spanning three pages of `UtxoOwners`. - let rows = MAX_REAPPLY_BATCH * 2 + 5; - let entries = [(nonce(1), 100u128), (nonce(2), 250u128)]; - PreForkStateKey::::put(seed_pre_fork_state(&entries)); - for i in 0..rows { - UtxoOwners::::insert(H256::from_low_u64_be(i as u64 + 1), owner_bytes()); - } - for (nonce, _) in entries.iter() { - UtxoOwners::::insert(nonce, owner_bytes()); - } - let total = UtxoOwners::::iter().count() as u32; - - assert_eq!(run_to_completion(), 4, "three pages plus the completing step"); - - let Some(Event::DustReapplyCompleted { applied, skipped }) = cnight_events().pop() else { - panic!("replay must complete, got {:?}", cnight_events()); - }; - assert_eq!(applied, 2); - assert_eq!(applied + skipped, total, "every row must be visited exactly once"); - assert_eq!(Pallet::::on_chain_storage_version(), 2); - }); -} - -/// A batch that fails *after* something has already been restored is a genuine -/// batch failure, not the "dust survived the hardfork" signal: report its nonces -/// and carry on to the next page. -#[test] -fn later_batch_failure_is_reported_and_the_replay_completes() { - new_test_ext().execute_with(|| { - init_ledger_state(); - - let pre_fork_key = seed_pre_fork_state(&[(nonce(1), 100u128)]); - PreForkStateKey::::put(pre_fork_key.clone()); - UtxoOwners::::insert(nonce(1), owner_bytes()); - run_to_completion(); - - // Replay the same nonce again — it is now present in the ledger, so its - // batch fails — but against progress that says an earlier page landed. - frame_system::Pallet::::reset_events(); - StorageVersion::new(1).put::(); - PreForkStateKey::::put(pre_fork_key); - DustReapplyProgress::::put((5, 0)); - - assert_eq!(run_to_completion(), 2, "the replay must carry on past a failed batch"); - - assert_eq!( - cnight_events(), - vec![ - Event::DustReapplyBatchFailed { nonces: vec![nonce(1)] }, - Event::DustReapplyCompleted { applied: 5, skipped: 1 }, - ], - ); - assert_eq!(Pallet::::on_chain_storage_version(), 2); - }); -} - -/// A `PreForkStateKey` that isn't a ledger-8 root (here: the current v9 root) -/// must abandon the replay rather than silently restore nothing. -#[test] -fn unreadable_pre_fork_key_cancels() { - new_test_ext().execute_with(|| { - init_ledger_state(); - - PreForkStateKey::::put(mock::Midnight::state_key()); - UtxoOwners::::insert(nonce(1), owner_bytes()); - - assert_eq!(run_to_completion(), 1); - - assert_eq!(cnight_events(), vec![Event::DustReapplySkipped]); - assert_eq!(Pallet::::on_chain_storage_version(), 2); - assert!(applied_dust_events().is_empty()); - }); -} diff --git a/pallets/cnight-observation/tests/migration_tests.rs b/pallets/cnight-observation/tests/migration_tests.rs index 90fdfc235..4da2fb509 100644 --- a/pallets/cnight-observation/tests/migration_tests.rs +++ b/pallets/cnight-observation/tests/migration_tests.rs @@ -11,31 +11,24 @@ // See the License for the specific language governing permissions and // limitations under the License. -//! Storage migration tests that need no ledger. +//! v0 -> v1 storage migration tests. //! //! Drives `SteppedMigration::step` directly on the mock runtime; the MBM //! framework is not exercised here. Uses `mock_with_capture` to avoid the -//! ledger dependency — so this covers the v0 -> v1 migration in full, and the -//! parts of the v1 -> v2 dust replay that stop short of a ledger read (the rest -//! lives in `dust_reapply_tests.rs`, against a real ledger). +//! ledger dependency — the migration only touches pallet storage. use frame_support::{ migrations::{SteppedMigration, SteppedMigrationError}, pallet_prelude::*, storage_alias, - traits::OnRuntimeUpgrade, weights::{RuntimeDbWeight, WeightMeter}, }; use midnight_primitives_cnight_observation::{CardanoRewardAddressBytes, DustPublicKeyBytes}; use pallet_cnight_observation::{ - Config, DustReapplyCtime, DustReapplyProgress, Event, Mapping, MappingEntry, Pallet, - PreForkStateKey, + Config, Mapping, MappingEntry, Pallet, migrations::v1::{MAX_ENTRIES_PER_ADDR, MigrateV0ToV1}, - migrations::v2::{MigrateV1ToV2, RecordPreForkState}, -}; -use pallet_cnight_observation_mock::mock_with_capture::{ - MockLedgerStateKey, RuntimeEvent, System, Test, new_test_ext, }; +use pallet_cnight_observation_mock::mock_with_capture::{Test, new_test_ext}; use sidechain_domain::UtxoId; /// Matches the legacy pre-migration `Mappings` storage. Kept in a sub-module @@ -174,92 +167,6 @@ fn returns_cursor_to_resume_when_meter_exhausts_mid_migration() { }); } -fn cnight_events() -> Vec> { - System::events() - .iter() - .filter_map(|record| match &record.event { - RuntimeEvent::CNightObservation(e) => Some(e.clone()), - _ => None, - }) - .collect() -} - -/// The upgrade block must capture the (still untranslated) ledger-8 state key, -/// which is the only place the wiped dust entries' values and owners survive. -#[test] -fn records_pre_fork_state_key_on_upgrade() { - new_test_ext().execute_with(|| { - MockLedgerStateKey::set(vec![0xAB; 64]); - StorageVersion::new(1).put::>(); - - RecordPreForkState::::on_runtime_upgrade(); - - assert_eq!(PreForkStateKey::::get(), Some(vec![0xAB; 64])); - assert_eq!(cnight_events(), vec![Event::DustReapplyStarted]); - }); -} - -/// Already-migrated chains (and fresh ledger-9 genesis) must not re-arm it. -#[test] -fn records_nothing_when_already_at_v2() { - new_test_ext().execute_with(|| { - MockLedgerStateKey::set(vec![0xAB; 64]); - StorageVersion::new(2).put::>(); - - RecordPreForkState::::on_runtime_upgrade(); - - assert!(PreForkStateKey::::get().is_none()); - assert!(cnight_events().is_empty()); - }); -} - -/// The replay is deliberately paced at one batch per block by charging half a -/// block per step (the benchmarked `process_tokens` weight says nothing about the -/// ledger cost of 200 dust `Create`s). Pin that: anything less than half a block -/// must defer to the next block rather than run a second batch in this one. -#[test] -fn replay_step_charges_half_a_block() { - new_test_ext().execute_with(|| { - let block_weights: frame_system::limits::BlockWeights = - ::BlockWeights::get(); - let half_block = block_weights.max_block.ref_time() / 2; - - let mut meter = WeightMeter::with_limit(Weight::from_parts(half_block - 1, u64::MAX)); - assert!( - matches!( - MigrateV1ToV2::::step(None, &mut meter), - Err(SteppedMigrationError::InsufficientWeight { .. }) - ), - "under half a block, the step must defer", - ); - - let mut meter = WeightMeter::with_limit(Weight::from_parts(half_block, u64::MAX)); - assert!(MigrateV1ToV2::::step(None, &mut meter).is_ok()); - assert!( - meter.remaining().ref_time() < half_block, - "the step must consume what it charged, so no second batch fits", - ); - }); -} - -/// Without a pre-fork state key there is nothing to replay (no v8 fork happened), -/// so the migration must wind itself up rather than stall the observer. -#[test] -fn replay_without_pre_fork_key_cancels() { - new_test_ext().execute_with(|| { - StorageVersion::new(1).put::>(); - DustReapplyProgress::::put((5, 5)); - - let mut meter = WeightMeter::new(); - assert!(MigrateV1ToV2::::step(None, &mut meter).unwrap().is_none()); - - assert_eq!(cnight_events(), vec![Event::DustReapplySkipped]); - assert_eq!(Pallet::::on_chain_storage_version(), 2); - assert!(DustReapplyCtime::::get().is_none()); - assert_eq!(DustReapplyProgress::::get(), (0, 0)); - }); -} - #[test] fn step_resumes_strictly_past_provided_cursor() { new_test_ext().execute_with(|| { diff --git a/pallets/cnight-observation/tests/tests.rs b/pallets/cnight-observation/tests/tests.rs index e29d8309a..329421e7d 100644 --- a/pallets/cnight-observation/tests/tests.rs +++ b/pallets/cnight-observation/tests/tests.rs @@ -1541,20 +1541,18 @@ fn position_guard_works_with_utxos_present() { }); } -/// While any MBM of this pallet's storage is still draining, `process_tokens` -/// must short-circuit: mid v0 -> v1 it would read only v1 and silently corrupt -/// registration state for any reward address whose v0 row hasn't been moved yet -/// (e.g. a deregistration would no-op here and then re-appear as live once the -/// migration completes); mid v1 -> v2 it would race the dust generation replay. +/// While the v0 -> v1 MBM is still draining, `process_tokens` must short-circuit: +/// reading only v1 mid-migration would silently corrupt registration state for any +/// reward address whose v0 row hasn't been moved yet (e.g. a deregistration would +/// no-op here and then re-appear as live once the migration completes). /// -/// This test forces `on_chain_storage_version` back to 0, then to 1, to simulate -/// blocks where either MBM is mid-flight, and asserts that an inherent carrying -/// real UTXOs: +/// This test forces `on_chain_storage_version` back to 0 to simulate a block where +/// the MBM is mid-flight, then asserts that an inherent carrying real UTXOs: /// - leaves `NextCardanoPosition` unchanged, /// - writes nothing to `Mapping`, /// - emits no pallet events. -/// After flipping the version to 2 (both migrations complete), the same call -/// processes normally and updates state. +/// After flipping the version to 1 (migration complete), the same call processes +/// normally and updates state. #[test] fn process_tokens_skips_during_mbm_then_resumes() { new_test_ext().execute_with(|| { @@ -1610,30 +1608,9 @@ fn process_tokens_skips_during_mbm_then_resumes() { advance_block_and_reset_events(); - // Same at version 1, where the v1 -> v2 dust generation replay is the - // migration in flight. - StorageVersion::new(1).put::(); - - let inherent = create_inherent(utxos.clone(), test_position(10, 1)); - let call = CNightObservation::create_inherent(&inherent).unwrap(); - assert_ok!(RuntimeCall::CNightObservation(call).dispatch(RawOrigin::None.into())); - - assert_eq!( - NextCardanoPosition::::get(), - position_before, - "NextCardanoPosition must not advance during the v1 -> v2 MBM", - ); - assert_eq!( - Mapping::::iter_prefix_values(cardano_reward_address).count(), - 0, - "no Mapping rows must be written during the v1 -> v2 MBM", - ); - - advance_block_and_reset_events(); - - // Migrations complete: storage version flips to 2; the next inherent + // Migration completes: storage version flips to 1; the next inherent // processes the same UTXOs normally. - StorageVersion::new(2).put::(); + StorageVersion::new(1).put::(); let inherent = create_inherent(utxos, test_position(10, 1)); let call = CNightObservation::create_inherent(&inherent).unwrap(); diff --git a/pallets/midnight/src/lib.rs b/pallets/midnight/src/lib.rs index e6b67aa18..8a6c36225 100644 --- a/pallets/midnight/src/lib.rs +++ b/pallets/midnight/src/lib.rs @@ -25,7 +25,7 @@ mod runtime_api; pub use runtime_api::*; pub use midnight_primitives::{ - LedgerMutFn, LedgerStateProvider, LedgerStateProviderMut, TransactionType, TransactionTypeV2, + LedgerMutFn, LedgerStateProviderMut, TransactionType, TransactionTypeV2, }; pub use midnight_node_ledger::types::active_version::LedgerApiError; @@ -60,13 +60,11 @@ pub mod pallet { }; use sp_runtime::Weight; - impl super::LedgerStateProvider for Pallet { + impl super::LedgerStateProviderMut for Pallet { fn get_ledger_state_key() -> Vec { StateKey::::get() } - } - impl super::LedgerStateProviderMut for Pallet { #[allow(clippy::unwrap_in_result)] // generic error type E cannot be constructed here fn mut_ledger_state(f: F) -> Result where diff --git a/primitives/midnight/src/lib.rs b/primitives/midnight/src/lib.rs index 8bfc2cd3b..c71d36c79 100644 --- a/primitives/midnight/src/lib.rs +++ b/primitives/midnight/src/lib.rs @@ -26,15 +26,10 @@ use scale_info::TypeInfo; use sp_runtime::DispatchError; pub type LedgerMutFn = fn(Vec) -> Result, E>; - -/// Trait to allow pallets to read the current Ledger state key -pub trait LedgerStateProvider { - /// Get the current ledger state key - fn get_ledger_state_key() -> Vec; -} - /// Trait to allow pallets to mutate the Ledger state pub trait LedgerStateProviderMut { + /// Get the current ledger state key + fn get_ledger_state_key() -> Vec; /// Mutate the ledger state - must return an updated ledger state key and may optionally return extra data fn mut_ledger_state(f: F) -> Result where diff --git a/runtime/src/lib.rs b/runtime/src/lib.rs index e54dd7b5d..550e220df 100644 --- a/runtime/src/lib.rs +++ b/runtime/src/lib.rs @@ -488,11 +488,7 @@ parameter_types! { impl pallet_migrations::Config for Runtime { type RuntimeEvent = RuntimeEvent; #[cfg(not(feature = "runtime-benchmarks"))] - // Append-only: `ActiveCursor.index` indexes this tuple. - type Migrations = ( - pallet_cnight_observation::migrations::v1::MigrateV0ToV1, - pallet_cnight_observation::migrations::v2::MigrateV1ToV2, - ); + type Migrations = (pallet_cnight_observation::migrations::v1::MigrateV0ToV1,); // Benchmarks need mocked migrations to guarantee that they succeed. #[cfg(feature = "runtime-benchmarks")] type Migrations = pallet_migrations::mock_helpers::MockedMigrations; @@ -903,8 +899,6 @@ parameter_types! { impl pallet_cnight_observation::Config for Runtime { type MidnightSystemTransactionExecutor = MidnightSystem; - type LedgerStateProvider = Midnight; - type LedgerBlockContextProvider = Midnight; type WeightInfo = weights::pallet_cnight_observation::WeightInfo; } @@ -1127,11 +1121,6 @@ pub type CheckedExtrinsic = generic::CheckedExtrinsic, - // MUST precede the pallet-midnight translation below: it captures the - // still-untranslated v8 state key, which the cNIGHT dust generation replay - // (`pallet_cnight_observation::migrations::v2::MigrateV1ToV2`) reads the - // wiped entries' values and owners from. - pallet_cnight_observation::migrations::v2::RecordPreForkState, // Ledger v8 -> v9 state translation (the ledger 8->9 hardfork). Runs once, // when a ledger-8 runtime (pallet-midnight storage version 1) upgrades to // this ledger-9 runtime (storage version 2). diff --git a/util/toolkit/tests/hardfork_e2e.rs b/util/toolkit/tests/hardfork_e2e.rs index 86b7f1138..da7d11a63 100644 --- a/util/toolkit/tests/hardfork_e2e.rs +++ b/util/toolkit/tests/hardfork_e2e.rs @@ -26,9 +26,6 @@ use testcontainers::{ runners::AsyncRunner, }; -/// Genesis-funded dev wallet the test transacts from. -const SOURCE_SEED: &str = "0000000000000000000000000000000000000000000000000000000000000001"; - /// Generate a chain-spec JSON string by running `build-spec` in the fork-from node container. fn generate_chainspec(image: &str, tag: &str) -> String { let output = Command::new("docker") @@ -127,20 +124,6 @@ async fn find_code_applied_block(rpc: &RpcClient, head: u64, old_spec: u64) -> u lo } -/// A plain (non-map) storage value at `hash`, or `None` if unset. -async fn storage_at(rpc: &RpcClient, pallet: &[u8], item: &[u8], hash: &str) -> Option> { - let key = format!( - "0x{}{}", - hex::encode(sp_crypto_hashing::twox_128(pallet)), - hex::encode(sp_crypto_hashing::twox_128(item)), - ); - let value: Option = rpc - .request("state_getStorage", rpc_params![&key, hash]) - .await - .unwrap_or_else(|e| panic!("state_getStorage({key}) failed at {hash}: {e}")); - value.map(|v| hex::decode(v.trim_start_matches("0x")).expect("hex-encoded storage value")) -} - /// Every way of reading the ledger state must answer at `height`. /// /// Both the `midnight_*` RPCs and a raw `state_call`: the fix lives in the ledger-9 @@ -212,7 +195,7 @@ async fn hardfork_single_tx() { "inmemory", "single-tx", "--source-seed", - SOURCE_SEED, + "0000000000000000000000000000000000000000000000000000000000000001", "--unshielded-amount", "10", "--destination-address", @@ -290,61 +273,6 @@ async fn hardfork_single_tx() { assert_ledger_state_readable(&rpc, applied, "code-applied block").await; assert_ledger_state_readable(&rpc, applied + 1, "post-migration").await; - // 5b. The cNIGHT dust generation replay (pallet-cnight-observation v1 -> v2) - // arms itself in the code-applying block and then runs as a multi-block - // migration. It must wind up: while it is in flight `process_tokens` - // ignores every Cardano observation, so a replay that never finishes - // silently strands the observer. Storage version 2 with the pre-fork key - // cleared is exactly "wound up", by either the restore or the - // self-cancel path. - // - // The `dev` preset carries no `UtxoOwners` rows, so this exercises the - // arming and wind-up, not the restore. Nor is the self-cancel visible - // here for the same reason (with nothing to replay there is no colliding - // `Create`); the pallet tests cover both against a real ledger state. - wait_for_finalized_block(&url, applied + 3, Duration::from_secs(60)).await; - let head_hash = block_hash_at(&rpc, applied + 3).await; - assert_eq!( - storage_at(&rpc, b"CNightObservation", b":__STORAGE_VERSION__", &head_hash).await, - Some(vec![2, 0]), - "cnight-observation must reach storage version 2 (dust replay wound up) by #{}", - applied + 3, - ); - assert_eq!( - storage_at(&rpc, b"CNightObservation", b"PreForkStateKey", &head_hash).await, - None, - "the pre-fork ledger state key must be cleared once the dust replay winds up", - ); - eprintln!("[hardfork_e2e] dust generation replay wound up by #{}", applied + 3); - - // 5c. The fork wipes dust state, and the `dev` preset has no `UtxoOwners` for - // the replay above to restore, so the genesis wallets cross the fork still - // holding NIGHT but generating no DUST — and with no DUST they cannot pay - // a fee. Re-register the source wallet's dust address to start generation - // again. The registration funds itself from the retroactive DUST its - // now-generationless NIGHT accrued, which is exactly the path a real - // holder takes after the wipe. - run_cli(&[ - "generate-txs", - "--fetch-cache", - "inmemory", - "register-dust-address", - "--wallet-seed", - SOURCE_SEED, - "-s", - &url, - "-d", - &url, - ]) - .await; - - // The sender only returns once the registration is finalized, but the NIGHT it - // re-registered starts generating from *that* block's time — at the tip there - // is still nothing accrued to spend. Give it a couple of blocks. - let registered_at = finalized_height(&rpc).await; - wait_for_finalized_block(&url, registered_at + 2, Duration::from_secs(60)).await; - eprintln!("[hardfork_e2e] dust address re-registered by #{registered_at}"); - // 6. Post-fork: run single-tx again to verify the node still works after the (future) upgrade run_cli(&[ "generate-txs", @@ -352,7 +280,7 @@ async fn hardfork_single_tx() { "inmemory", "single-tx", "--source-seed", - SOURCE_SEED, + "0000000000000000000000000000000000000000000000000000000000000001", "--unshielded-amount", "10", "--destination-address", From 7736a887e46b67b2f690ad6907eff1b214d7d10b Mon Sep 17 00:00:00 2001 From: Oscar Bailey <79094698+ozgb@users.noreply.github.com> Date: Thu, 13 Aug 2026 11:49:03 +0100 Subject: [PATCH 11/13] chore: rebuild metadata Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> --- metadata/static/midnight_metadata.scale | Bin 136548 -> 135037 bytes metadata/static/midnight_metadata_2.1.0.scale | Bin 136548 -> 135037 bytes 2 files changed, 0 insertions(+), 0 deletions(-) diff --git a/metadata/static/midnight_metadata.scale b/metadata/static/midnight_metadata.scale index 49e1f4cdf87b62825cf07691fa85cd4c116097fc..619fd321e70b4c53ebae5e1462e14d89550a2a44 100644 GIT binary patch delta 38 wcmV+>0NMZKstEm_2(aAQ0Th$n*~pV&(jl|j-OR-SB)6xd0XYJ$!fgFc5T-;)5Y{uHB~h; zwwLyQ*qlW?dh*91e}N+C!D~b>l4Je?iQuc6o?#Xb&Shq*`>XH0_r3T1^JVYJH@$EF zcq}fT{`vUl_F#YJ!#$RDT{ZWEbb(d&eDCQ|$y7?GM%GPeR*#)jOed8sCM4aI6$QMY zP)cJq>}1f^kmI_N&C{KHs!kq)4F2l)e{$D z8_nvP*q*(4H4_*7MA!BD|BJ7@ETW)Yi;*_km$7dy+ht)+Y=YCVYlN4e0oe`8yPNX# z;7P>2GsCO0n6T=TtQurlod6ZVm`OM5P0b2WT^<<&z&V0!@eZ2l`?WwhwXdyk6EUZ(VE}@hmx{RQ~ z%tYnnt0xtw`S0VehllZ~vg7gTcYk00;iun!{ZU+bbo!OJ{#<+W(D5y#0U-%|o0}{< zzm>hTE24R5soCB&goBMFXE7&q?-X4VI1{8kzy+xdwDC3)eN8MS?`(25Ms3PbA!Ev< zqDVK%fEkv7=CX>Gl}3?WAdmi{AUO(%R$j{rEZ|D*dygaX3yf?XqXBq`A6>})%r2Tq3o!d;({ zc36MiuzcWpkloo}edz2MIrbv^R8ZD^dT$82H!~(~!kunu^r_4EE(OdgE`iQBiAmSs rK@44_ml9~rUQxn3LV%c7jMWGRz@6q;EH@MO@BdHg(Jx<%@2h_SS&S?} diff --git a/metadata/static/midnight_metadata_2.1.0.scale b/metadata/static/midnight_metadata_2.1.0.scale index 49e1f4cdf87b62825cf07691fa85cd4c116097fc..619fd321e70b4c53ebae5e1462e14d89550a2a44 100644 GIT binary patch delta 38 wcmV+>0NMZKstEm_2(aAQ0Th$n*~pV&(jl|j-OR-SB)6xd0XYJ$!fgFc5T-;)5Y{uHB~h; zwwLyQ*qlW?dh*91e}N+C!D~b>l4Je?iQuc6o?#Xb&Shq*`>XH0_r3T1^JVYJH@$EF zcq}fT{`vUl_F#YJ!#$RDT{ZWEbb(d&eDCQ|$y7?GM%GPeR*#)jOed8sCM4aI6$QMY zP)cJq>}1f^kmI_N&C{KHs!kq)4F2l)e{$D z8_nvP*q*(4H4_*7MA!BD|BJ7@ETW)Yi;*_km$7dy+ht)+Y=YCVYlN4e0oe`8yPNX# z;7P>2GsCO0n6T=TtQurlod6ZVm`OM5P0b2WT^<<&z&V0!@eZ2l`?WwhwXdyk6EUZ(VE}@hmx{RQ~ z%tYnnt0xtw`S0VehllZ~vg7gTcYk00;iun!{ZU+bbo!OJ{#<+W(D5y#0U-%|o0}{< zzm>hTE24R5soCB&goBMFXE7&q?-X4VI1{8kzy+xdwDC3)eN8MS?`(25Ms3PbA!Ev< zqDVK%fEkv7=CX>Gl}3?WAdmi{AUO(%R$j{rEZ|D*dygaX3yf?XqXBq`A6>})%r2Tq3o!d;({ zc36MiuzcWpkloo}edz2MIrbv^R8ZD^dT$82H!~(~!kunu^r_4EE(OdgE`iQBiAmSs rK@44_ml9~rUQxn3LV%c7jMWGRz@6q;EH@MO@BdHg(Jx<%@2h_SS&S?} From 72d4140170ff6a71aadb94badc3b5621d73ac898 Mon Sep 17 00:00:00 2001 From: Oscar Bailey <79094698+ozgb@users.noreply.github.com> Date: Thu, 13 Aug 2026 12:41:31 +0100 Subject: [PATCH 12/13] chore: bump metadata package version Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> --- Cargo.lock | 2 +- metadata/Cargo.toml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 1c9424d93..2cef37de2 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -8127,7 +8127,7 @@ dependencies = [ [[package]] name = "midnight-node-metadata" -version = "2.0.0" +version = "2.1.0" dependencies = [ "subxt 0.50.0", "walkdir", diff --git a/metadata/Cargo.toml b/metadata/Cargo.toml index 627859ed4..635b0e763 100644 --- a/metadata/Cargo.toml +++ b/metadata/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "midnight-node-metadata" -version = "2.0.0" +version = "2.1.0" edition = "2024" build = "build.rs" license-file.workspace = true From 5da44a54e603c510a47adb37f379373e092776c3 Mon Sep 17 00:00:00 2001 From: Oscar Bailey <79094698+ozgb@users.noreply.github.com> Date: Thu, 23 Jul 2026 12:19:13 +0000 Subject: [PATCH 13/13] chore: update npm to fix toolkit SBOM error (#1919) * chore: update npm to fix toolkit SBOM error Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> * docs: add change file Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> --------- Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> --- Earthfile | 8 ++++---- .../changed/bump-npm-sbom-tar-critical.md | 17 +++++++++++++++++ 2 files changed, 21 insertions(+), 4 deletions(-) create mode 100644 changes/toolkit/changed/bump-npm-sbom-tar-critical.md diff --git a/Earthfile b/Earthfile index 77360edd5..9dbb118de 100644 --- a/Earthfile +++ b/Earthfile @@ -1415,7 +1415,7 @@ toolkit-image: tar -xJf node.tar.xz -C /usr/local --strip-components=1 && \ rm node.tar.xz && \ node --version && npm --version && \ - npm install -g npm@11.11.0 && npm --version + npm install -g npm@11.18.0 && npm --version # Add toolkit-js (only when INCLUDE_TOOLKIT_JS=true) IF [ "$INCLUDE_TOOLKIT_JS" = "true" ] @@ -1475,7 +1475,7 @@ audit-npm: curl -fsSL https://nodejs.org/dist/v${NODE_VERSION}/node-v${NODE_VERSION}-linux-${NODE_ARCH}.tar.xz -o node.tar.xz && \ tar -xJf node.tar.xz -C /usr/local --strip-components=1 && \ rm node.tar.xz && \ - npm install -g npm@11.11.0 && \ + npm install -g npm@11.18.0 && \ node --version && npm --version COPY ${DIRECTORY} ${DIRECTORY} @@ -1514,7 +1514,7 @@ audit-yarn: curl -fsSL https://nodejs.org/dist/v${NODE_VERSION}/node-v${NODE_VERSION}-linux-${NODE_ARCH}.tar.xz -o node.tar.xz && \ tar -xJf node.tar.xz -C /usr/local --strip-components=1 && \ rm node.tar.xz && \ - npm install -g npm@11.11.0 && \ + npm install -g npm@11.18.0 && \ node --version && npm --version # Install and enable corepack for yarn support @@ -1565,7 +1565,7 @@ fix-lock-npm: curl -fsSL https://nodejs.org/dist/v${NODE_VERSION}/node-v${NODE_VERSION}-linux-${NODE_ARCH}.tar.xz -o node.tar.xz && \ tar -xJf node.tar.xz -C /usr/local --strip-components=1 && \ rm node.tar.xz && \ - npm install -g npm@11.11.0 && \ + npm install -g npm@11.18.0 && \ node --version && npm --version COPY ${DIRECTORY}/package.json ${DIRECTORY}/package-lock.json ${DIRECTORY}/ diff --git a/changes/toolkit/changed/bump-npm-sbom-tar-critical.md b/changes/toolkit/changed/bump-npm-sbom-tar-critical.md new file mode 100644 index 000000000..0af6f436e --- /dev/null +++ b/changes/toolkit/changed/bump-npm-sbom-tar-critical.md @@ -0,0 +1,17 @@ +#toolkit #security +# Bump bundled npm 11.11.0 -> 11.18.0 to clear toolkit image SBOM findings + +The Grype scan of the toolkit image failed on a critical `tar` advisory +(GHSA-23hp-3jrh-7fpw, `tar@7.5.9`, fixed in 7.5.19). That `tar`, along with the +other flagged npm packages (sigstore, @sigstore/core, @sigstore/verify, +minimatch, brace-expansion, picomatch, ip-address), is vendored inside the +globally-installed npm CLI, not in toolkit-js's dependencies. + +- Bumped the pinned `npm install -g npm@11.11.0` to `npm@11.18.0` across the + Earthfile targets (`toolkit-image`, `audit-npm`, `audit-yarn`, + `fix-lock-npm`). npm 11.18.0 bundles `tar@7.5.19` (clearing the critical) plus + patched versions of every other flagged npm package. +- Minor bump within the 11.x line; engine requirement is unchanged + (`^20.17.0 || >=22.9.0`), satisfied by the image's Node 24.18.0. + +PR: https://github.com/midnightntwrk/midnight-node/pull/1919