feat(ledger): backport the ledger 8->9 hardfork to release/node-2.1.0 - #2019
feat(ledger): backport the ledger 8->9 hardfork to release/node-2.1.0#2019ozgb wants to merge 13 commits into
Conversation
* feat(ledger): add v8->v9 storage migration for the ledger 8->9 hardfork Port the v8->v9 state translation table from midnight-ledger PR #539 into `midnight_node_ledger::state_translation_v8_to_v9`, and wire it into the runtime as a single-block storage migration: - New host function `Ledger9Bridge::migrate_state_v8_to_v9` reads the v8 arena root (pallet-midnight `StateKey`), walks/translates the v8 `LedgerState` into the v9 shape, re-persists it, and returns the new v9 root. v8 and v9 share one storage backend, so the arena is shared. - `pallet_midnight::migrations::v2::MigrateV1ToV2` (VersionedMigration 1->2) calls the host function and re-points `StateKey`. Bumped pallet-midnight STORAGE_VERSION 1 -> 2 and added it to the runtime `Migrations` tuple. - Un-ignore the `hardfork_single_tx` e2e test and point its fork-from image at the ledger-8 release `midnightntwrk/midnight-node:1.0.1`. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> * test(ledger): add v8->v9 state translation table + smoke tests - table_is_closed / table_tags_match_types guard the translation table against tag drift on the node's rc.3 crate versions. - empty_state_translates_and_round_trips exercises an end-to-end v8->v9 translation and a v9 serialize round-trip. - Enable the helpers `can-panic` feature for ledger dev builds so the crate's `#[cfg(test)]` modules compile standalone. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> * fix(node): version-aware ledger genesis seeding for the 8->9 hardfork A ledger-9 node booting on a ledger-8 chain-spec (the hardfork fork-from case) panicked at startup: the arena seeder hardcoded the ledger-9 deserializer and rejected the v8 genesis (`expected ledger-state[v18], got ledger-state[v13]`). The genesis block runs under the old WASM, so the arena must be seeded in the genesis version. - Add `genesis_matches_this_version` (per-version tag check) to common storage. - Add `init_ledger_storage_{separate,unified}` dispatchers that pick the ledger_8 vs ledger_9 seeder by the genesis `ledger-state[vN]` tag; v8 and v9 share one backend so a v8-seeded arena is what the post-migration v9 reads. - Route `custom_parity_db` through the dispatchers. - Add a dev test proving the v8-seeded root matches the fork-from chain-spec's genesisStateKey (no `Ledger`-wrapper drift vs release 1.0.1). Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> * refactor(ledger): use serialize peek_tag for genesis version detection Replace the substring header scan in `genesis_matches_this_version` with `midnight_serialize::peek_tag`, matching the pattern used by `contract_operation_versioned_verifier_key`. Compares the peeked header tag against this version's `LedgerState::tag()` (no hardcoded version numbers). Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> * fix(ledger): restore ledger-8 construct_distribute_treasury host fn A ledger-9 node could not instantiate the ledger-8 runtime WASM (the hardfork fork-from case): the WASM imports `ext_ledger_8_bridge_construct_distribute_treasury_system_tx_version_1`, which was dropped from the current ledger-8 bridge (renamed to reserve/unlock-to-treasury for v9). Re-add it so the current node can execute the ledger-8 runtime across the 8->9 boundary. - ledger_8 builds `SystemTransaction::PayBlockRewardsToTreasury { amount }` (matching release 1.0.1); v7/v9 helpers are error stubs (only the ledger-8 bridge exposes the host fn). - Wire through the common Bridge and the Ledger8Bridge runtime interface. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> * fix(toolkit): runtime-upgrade robust to the ledger 8->9 metadata switch `runtime-upgrade` used subxt's `wait_for_finalized_success()`, which eagerly decodes the apply block's events. Across the 8->9 hardfork the client's metadata follows the code swap to the new runtime, so decoding the old runtime's `System.CodeUpdated` event fails ("Can't decode field hash ..."). Wait only for finalization (`wait_for_finalized`, no event decode) and confirm the upgrade enacted by polling `state_getRuntimeVersion` for the spec_version bump — no metadata-dependent decoding across the boundary. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> * feat(toolkit): real v8->v9 fork translation at the tx-generation boundary The toolkit built a ledger-8 transaction for the post-hardfork (ledger-9) chain, which the node rejected (Deserialization(Transaction): transaction[v9]/signature[v1] vs expected transaction[v12]/signature[v2]). Its fork-aware replay never transitioned the context ledger-8 -> ledger-9. - Move StateTranslationTable into `midnight-node-ledger-helpers` (so both the runtime migration in `ledger` and the toolkit fork can use it), adding the onchain-state deps there. - `fork_context_8_to_9` now runs the real `TypedTranslationState` translation (Db8 == Db9, one shared arena) instead of the tag-reuse `old_to_new_sp`. - Wire the ledger-8 -> ledger-9 transition into `replay_blocks` (new `fork_8_to_9_if_needed`) and relax the "not supported yet" assert. - runtime-upgrade now waits for the spec bump at the FINALIZED head (toolkit fetch reads only finalized blocks), so the post-fork fetch sees the ledger-9 blocks and the replay forks to ledger 9. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> * chore(toolkit): TEMP hardfork_debug log for replay_blocks partition Temporary diagnostic to see the l7/l8/l9 block partition + initial context version during the post-fork tx build. Remove once the fork boundary is fixed. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> * fix(toolkit): runtime-upgrade waits for the new runtime to EXECUTE, not just apply state_getRuntimeVersion(finalized) reports the *stored* code, which flips to the new runtime at the apply_authorized_upgrade block — but that block still *executes* under the old runtime (its MNSV digest, which the fetcher uses to classify ledger version, is the old spec). The first block to run the new runtime is apply+1. The prior poll returned at the apply block, so a downstream fetch (bounded by finalized height) reached the apply block (still classified ledger-8) but not apply+1, and the toolkit built a ledger-8 tx post-fork. Track the finalized height where the stored spec first bumps, then wait for the finalized height to advance past it (apply+1 finalized) before reporting success. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> * chore(toolkit): remove TEMP hardfork_debug replay_blocks log The v8->v9 fork boundary is fixed and hardfork_e2e passes; drop the diagnostic. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> * docs: add change files for the ledger 8->9 hardfork migration Assisted-by: Claude:claude-sonnet-5 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> * chore: cargo fmt Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> * chore: fix clippy errors Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> * chore: npm audit fix (toolkit-js) Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> * feat: add perf prints to track migration time elapsed Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> * feat: use ledger cost model for storage migration Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> * chore(local-environment): npm audit fix Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> * fix: no-op ledger v8->v9 migration when state is already ledger-9 The 2.0.0 runtime runs ledger-9 but shipped pallet-midnight at storage version 1 (it had no v1->v2 migration), so a network upgrading 2.0.0 -> this runtime still fires VersionedMigration<1,2> over an already-v9 StateKey. Feeding that v9 root to the v8 decode path fails on the tag mismatch and the pallet's expect() panics, bricking the upgrade block. Guard migrate_state_v8_to_v9 so it no-ops when the StateKey already references a ledger-9 state: detect it by the arena root's serialized tag (distinct between ledger-state[v13] and [v18]) and return the key unchanged with zero synthetic cost, leaving only the storage-version bump. Assisted-by: Claude:claude-fable-5 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> * docs: update change file Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> --------- Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> Co-authored-by: Squirrel <giles.cope@shielded.io> (cherry picked from commit 74a9115) Assisted-by: Claude:claude-opus-5 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
#1925 threads a `&WalletSchemes` through `fork_8_to_9_if_needed` and `replay_blocks_9`. That parameter comes from the ECDSA work on main (#1837, #1861), which is not on release/node-2.0.0, so the cherry-pick merges textually clean and only fails at compile. Drop the parameter and its three call sites; wallet-scheme selection does not exist on this branch and nothing else in the fork path reads it. Adaptation of the cherry-pick, not an upstream change. Assisted-by: Claude:claude-opus-5 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
Signed-off-by: Giles Cope <gilescope@gmail.com> Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> # Conflicts: # Cargo.lock # Cargo.toml Co-authored-by: Giles Cope <gilescope@gmail.com> (cherry picked from commit 6fb0cc0) Assisted-by: Claude:claude-opus-5 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
… block (#1985) * fix(ledger): serve ledger state reads at the ledger-hardfork set_code block On a chain that hardforks ledger 8 -> 9 via a governance `set_code`, exactly one historical block is permanently unreadable: every ledger state read at that hash fails with `Deserialization(TypedArenaKey)` (GH #1959). The pre-fork runtime ships `system_version: 1`, so `frame_system` overwrites `:code` *inside* the `set_code` block, while pallet-midnight's v8 -> v9 state translation only runs in the next block's `initialize_block`. That block's committed state therefore pairs ledger-9 `:code` with a ledger-8 `StateKey` forever, and reading it executes ledger-9 code against a ledger-8 arena root. The read-only accessors of the ledger-9 host API now check the tagged-serialization header of the `state_key` they are handed and, when it is a ledger-8 arena root, serve the read from the ledger-8 bridge — which is what the state at that block actually is. v8 and v9 share one storage crate and hence one arena, so this is a pure dispatch with no data movement. The `StateKey` tag is the signal rather than the pallet storage version, because the 2.0.0 runtime already ran ledger 9 while still reporting pallet-midnight storage version 1. Guarded accessors: get_contract_state, get_zswap_chain_state, get_zswap_state_root, get_ledger_state_root, get_ledger_parameters, get_c_to_m_bridge_min_amount, get_unclaimed_amount, get_bridge_receiving_amount. Placing the dispatch in the host function rather than in the node's JSON-RPC layer covers every caller at once: the `midnight_*` RPCs, `MidnightRuntimeApi` through `state_call`, and subxt-based tooling such as chain-indexer (GH #1969), none of which touch the node's RPC handlers. The transaction paths are deliberately left alone — at the skew block they concern ledger-9-format transactions that ledger-8 code cannot deserialize in any case, and they resolve on their own one block later; this is documented at `get_transaction_cost`. hardfork_e2e now walks the fork boundary and asserts both the `midnight_*` RPCs and a raw `state_call` answer at `applied - 1`, `applied` and `applied + 1`. Assisted-by: Claude:claude-opus-5 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> * chore: point change file at the new PR Assisted-by: Claude:claude-opus-5 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> * docs: tighten doc comment Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> * docs: newline Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> --------- Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> (cherry picked from commit 54dffa0) Assisted-by: Claude:claude-opus-5 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
) Squashed backport of the ozgb-cnight-mbm branch (PR #2012), which is still open on main. The ledger 8->9 state translation wipes the ledger's dust state; this rebuilds cNIGHT's slice of the dust generating set as a multi-block migration driven from `UtxoOwners` plus the retained v8 arena root, backdating each replayed ctime to the DUST cap. Squashed rather than cherry-picked commit-by-commit because the branch contains a merge from main (16f07a8) that would drag in unrelated changes. PR: #2012 Assisted-by: Claude:claude-opus-5 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
The change file came across from the still-open #2012, which left the `PR:` field as a placeholder. Assisted-by: Claude:claude-opus-5 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a5310e4685
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
This comment has been minimized.
This comment has been minimized.
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f6b1d1f801
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
Reverts a5310e4 ("feat: reset Dust state and re-apply cNight UTxOs during migration (#2012)") and its follow-up 5b6bdc1, backing the cNIGHT dust generation replay out of the ledger-hardfork backport. PR #2012 is still open on main, so it lands there rather than here. Note: `metadata/static/midnight_metadata*.scale` were regenerated in 2ff2b44 while the reverted storage items (`PreForkStateKey`, `DustReapplyCtime`, `DustReapplyProgress`) were present, so metadata needs a rebuild. Assisted-by: Claude:claude-opus-5 Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
* chore: update npm to fix toolkit SBOM error Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> * docs: add change file Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com> --------- Signed-off-by: Oscar Bailey <79094698+ozgb@users.noreply.github.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5da44a54e6
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| &mut self, | ||
| state_key: PassFatPointerAndRead<&[u8]>, | ||
| ) -> AllocateAndReturnByCodec<Result<Vec<u8>, LedgerApiError>> { | ||
| serve_pre_migration_v8_read!(*self, state_key, get_ledger_parameters(state_key)); |
There was a problem hiding this comment.
Keep ledger-version reads fork-aware
When this branch is taken at the ledger 8→9 code-applied block, get_ledger_parameters returns the ledger-8 bridge's v8-tagged bytes, but MidnightRuntimeApi_get_ledger_version / midnight_getLedgerVersion(at) still use the unchanged ledger-9 get_version() path because it has no state-key guard. Callers that query the version API at that same hash to choose decoders will be told ledger 9 and then fail to decode these v8-tagged responses; make version reporting inspect StateKey or the block's execution-version digest for the skew block too.
Useful? React with 👍 / 👎.
|
Closing in favour of #2023 |
Overview
Backports the ledger 8 -> 9 hardfork onto the 2.1.0 release line.
The auditors are reviewing the tip of the release branch, so this is a cherry-pick
rather than a merge from
main. That is the whole point of the shape of this PR:mainNo new external dependencies.
Cargo.lockmoves three patch versions and nothingelse enters the graph:
midnight-ledger8.1.0 → 8.1.1,midnight-storage2.0.1 → 2.0.2(gains the
state-translationfeature),midnight-zswap8.1.0 → 8.1.1.Feature-gating the Q3 work behind
#[cfg(feature = "audit-q3")]was considered andrejected — it would still put 526 files in front of the auditors, plus ~500 new attributes
and a second build configuration.
Commits
Three commits are verbatim cherry-picks (each carries
(cherry picked from commit …), sothey can be diffed against the version already reviewed on
main):66b6d3552e940464ff75019d450set_codeblock#2012 was picked, then reverted — it is not in this backport
a5310e468(squashed #2012 — reset Dust state and re-apply cNIGHT UTxOs) and itsfollow-up
5b6bdc1fcare still in the branch history, but6c9d80c39reverts both.#2012 is still open on
mainand lands there instead; this backport carries only theledger 8→9 state translation. Review the net diff, not commit-by-commit — the four
files #2012 added (
ledger/src/host_api/dust_generation.rs,pallets/cnight-observation/src/migrations/v2.rs, its tests, and its change file) donot exist at the tip.
Consequences of the revert, all reflected in the tree:
STORAGE_VERSIONstays at 1 — no v2 MBM, andprocess_tokensis not gated behind one.
RecordPreForkStateis gone fromMigrations/Executive; the runtime's only ledgermigration is
pallet_midnight::migrations::v2::MigrateV1ToV2.7736a887eto drop the three storage items and four eventsfeat: reset Dust state and re-apply cNight UTxOs during migration #2012 had added.
Deviations from the upstream PRs — please review these two closely
41196f7e3(its own commit) — feat(ledger): ledger 8->9 hardfork on-chain migration #1925 threads a&WalletSchemesthroughfork_8_to_9_if_needed/replay_blocks_9. That parameter comes from the ECDSA work onmain(feat(toolkit): ecdsa support #1837, feat(toolkit): ECDSA contract maintenance & deploy committees #1861), which is not on this branch. The cherry-pick merges textuallyclean and only fails at compile; the parameter and its three call sites are dropped.
75019d450— fix(ledger): serve ledger state reads at the ledger-hardfork set_code block #1985 adds aserve_pre_migration_v8_read!guard toget_bridge_receiving_amount. That host fn does not exist on this branch (it arrived viafeat(c-to-m bridge): benchmarks for the bridge #1635 / feat(runtime): add Op variant for claim bridge transfers #1727), so the hunk is dropped. The other seven guarded accessors are all present
and unchanged.
🗹 TODO before merging
spec_version→
002_001_000needs a matchingRuntimeVersion::V2_1_0arm inutil/toolkit/src/fetcher/runtimes.rs, which needs aMidnightMetadata2_1_0module,which needs
metadata/static/midnight_metadata_2.1.0.scale. Bumping onlyspec_versionleaves the toolkit fetcher rejecting
002_001_000blocks asUnsupportedBlockVersion.Sequence:
/bot rebuild-metadata, then one commit mirroring301ee5035.Without a
spec_versionbump the migration never fires on a live 2.0.0 chain.mainalready occupies the2.1.0name — itsmidnight_metadata_2.1.0.scaleis adifferent runtime (BABE, consensus-engine). Needs renaming as part of the
main→ 3.0.0renumber, or the two files collide by name and disagree by content.
2ff2b4406with feat: reset Dust state and re-apply cNight UTxOs during migration #2012 in the tree,then
7736a887eafter the revert.res/. pallet-midnightSTORAGE_VERSION1 → 2 only affects freshbuild_stategenesis; the raw stagenetchainspecs still encode version 1, so they take the migration path.
📌 Submission Checklist
git commit -s) for the DCO — all are also GPG-signedMigrateV1ToV2translates the on-chainLedgerStatev13 → v18 and.expect()s onfailure, aborting the upgrade rather than leaving a ledger-9 runtime pointing at a v8 root.
with the cherry-picks (the fourth left with the feat: reset Dust state and re-apply cNight UTxOs during migration #2012 revert)
🧪 Testing Evidence
On the post-revert tree:
cargo check --workspace --tests— cleanOn the earlier tree of
a5310e468(i.e. with #2012 still applied — the ledger 8→9 codeunder review here is unchanged since, but the cNIGHT tests it ran are gone):
cargo check --workspace --all-targets(SKIP_WASM_BUILD=1) — cleancargo test -p pallet-midnight -p pallet-cnight-observation -p midnight-node-ledger—168 passed, 0 failed, across 3 runs
Not yet verified:
cargo check. CI covers it.wasm-opt-sysdisk-quota error, not acode error. Needs a real
cargo build --release. CI covers this.hardfork_e2e— needs Docker plus the1.0.1fork-from image. feat(ledger): ledger 8->9 hardfork on-chain migration #1925 un-ignores this testand repoints the default
FORK_FROM_NODE_IMAGEfrom0.21.0to1.0.1; all the Earthfileplumbing was already present on the release branch.
migrate_state_v8_to_v9entry.One flake to be aware of:
pallet-midnight --libfailed once on a cold build, then passed onthree subsequent runs including the identical invocation. Likely the process-global ledger
arena. Worth confirming it also flakes on unmodified
release/node-2.0.0before chasing it.🔱 Fork Strategy
This is the fork strategy: a governance
set_codeon a ledger-8 chain, with the v8 → v9state translation running as a
VersionedMigration<1, 2>in the following block'sinitialize_block.Rebuilding the cNIGHT dust generating set behind it (#2012) is not part of this backport —
it needs to land separately, and until it does a forked chain comes out of the hardfork with
the ledger's dust state wiped.
Links
Backport of:
Deliberately excluded:
main) — applied asa5310e468, reverted in6c9d80c39Not intentionally closing any issue — the upstream PRs already did.
Assisted-by: Claude:claude-opus-5