[HF v8→v9] Upgrade-block events cannot be decoded with post-upgrade metadata (false 'CodeUpdated not found' on a successful upgrade; chain-indexer exits)

[jacek-kurkowski-shielded]

Context & versions

• Pre-fork: node-1.0.0 — mainnet parity (specVersion 1000000, midnight_ledgerVersion "=8.1.0").
• Post-fork: main @ dd9fba007 — node 2.1.0-dd9fba00, specVersion 2001000. Also seen on 74a91156b.
• Network: local-environment local-env, 5 validators, federated governance runtime upgrade (npm run governance-runtime-upgrade:local-env).
• Clients involved: subxt 0.50 (as used by chain-indexer via indexer-common), and @polkadot/api as used by local-environment/src/commands/federatedRuntimeUpgrade.ts.
• Frequency: intermittent — observed in 4 of 6 upgrade runs. Two runs on dd9fba007 printed Runtime upgrade completed successfully instead, which is consistent with a race on when the client refreshes metadata.

Steps to reproduce

1. Run a ledger-8 local-env chain (5 validators).
2. Apply the governance runtime upgrade to spec 2001000:

   npm run governance-runtime-upgrade:local-env -- \
     --wasm test/midnight_node_runtime.compact.compressed.wasm \
     --rpc-url ws://localhost:9933 \
     --council-uris "//Four" "//Five" --technical-uris "//One" "//Two" \
     --executor-uri "//One" --skip-run
   

3. Observe the tool's final step (it looks for System.CodeUpdated in the block that applied the upgrade). Repeat a few times — it does not fail every run.
4. Independently, decode the events of the upgrade block with a subxt 0.50 client, and with the same client forced to use the pre-upgrade metadata version.

Expected behavior

The events of the block that applied the runtime upgrade should be decodable, and a successful runtime upgrade should not be reported as a failure.

Actual behavior

A block's events are encoded by the runtime of its parent state, but both subxt 0.50 and polkadot-js resolve metadata at the block's own hash. At the upgrade block those disagree, so event decoding fails there — and only there.

Evidence from a standalone subxt 0.50 probe against the live chain (querying one block at a time):

block N-1 : OK (events decoded)
block N   : FAIL - Can't decode event topics: Not enough data to fill buffer
block N+1 : OK (events decoded)

same client, block N, forced to pre-upgrade metadata (v15 and v14): OK (events decoded)

In subxt 0.50.1 the spec version comes from Core_version at the block's own hash and the metadata from Metadata_metadata_at_version at that same hash (src/client/online_client.rs, at_block_hash_and_number), which is what produces the mismatch.

Consequence 1 — governance-runtime-upgrade reports a false failure. Its last step asserts System.CodeUpdated is present in the applying block (local-environment/src/commands/federatedRuntimeUpgrade.ts:63-67). polkadot-js fails to decode that block's events, so the assertion throws even though the upgrade executed:

RPC-CORE: getStorage ... Unable to decode storage system.events ::
  createType(Vec<FrameSystemEventRecord>) ... H256:: Expected input with 32 bytes (256 bits), found 21 bytes
system.applyAuthorizedUpgrade included in block 0x891b...
Error: Runtime upgrade executed but System.CodeUpdated event not found.

The 21-vs-32 byte mismatch is the shape change itself: the ledger-8 runtime's CodeUpdated has no fields, the ledger-9 one carries a hash. state_getRuntimeVersion confirms the upgrade did take effect (1000000 → 2001000) in every run, including the ones where the tool threw.

This is operationally risky: an operator sees a hard failure on a successful upgrade and may retry it. Suggested fix for the tool: confirm via the state_getRuntimeVersion spec-version change rather than by inspecting the applying block's events.

Consequence 2 — chain-indexer exits on that block. With a ledger-9-capable indexer build, the chain-indexer dies at the same height:

index_blocks_task failed: get next block from node: cannot get next event:
  Can't decode event topics: Not enough data to fill buffer

and, in another run,

Can't decode field "hash" in event System.CodeUpdated: Decode error: Not enough data to fill buffer

(The differing message between runs is the same root cause; which field trips first depends on the events present.)

Scope note: the underlying metadata-resolution behaviour sits in the client libraries (subxt, polkadot-js) rather than in this repo, so the actionable part here is (a) the governance-runtime-upgrade success check, and (b) deciding whether the node should expose something that lets clients resolve the correct metadata for an upgrade block. Worth raising the general case upstream separately.

[wbaxterh]

Happy to take this one — I'd like to fix Consequence 1 (the governance-runtime-upgrade false failure) in this repo.

Root cause (agreeing with the write-up): federatedRuntimeUpgrade.ts decides success by decoding the applying block's events and asserting System.CodeUpdated is present (local-environment/src/commands/federatedRuntimeUpgrade.ts:63-67, via hasEvent). At the upgrade block the events are encoded by the parent (pre-upgrade) runtime, but polkadot-js resolves metadata at the block's own (post-upgrade) hash, so the CodeUpdated shape change (v8: no fields → v9: carries a hash) makes that decode fail and read as absent even though the upgrade executed. That matches the intermittency (a race on when the client refreshes metadata) and the fact that state_getRuntimeVersion still shows 1000000 → 2001000 in every run.

Proposed fix (the approach you suggested): confirm the upgrade by the runtime version actually changing rather than by decoding that block's events:

• capture spec_version before applying, then read it at the applied block hash via state_getRuntimeVersion (which decodes Core_version, not the events storage, so it is immune to the mismatch);
• treat a changed spec_version as success;
• fall back to the existing CodeUpdated event check only when spec_version is unchanged (the --allow-same-version path, where the event shape cannot have changed and the decode is reliable).

federatedRuntimeUpgrade.ts is the only command doing this event-based success check, so the change is localized.

Scope: I'm treating Consequence 2 (subxt/polkadot-js resolving metadata at the block's own hash, and the chain-indexer exit) and the open "should the node expose something so clients can resolve the right metadata for an upgrade block" question as out of scope here / upstream, and leaving them for a separate discussion.

I'll open a PR shortly.

[gilescope]

The governance-runtime-upgrade tool asserts System.CodeUpdated is present in the applying block's events (federatedRuntimeUpgrade.ts:63), but polkadot-js resolves metadata at the block's own (post-upgrade) hash while those events were encoded by the parent (pre-upgrade) runtime; the field shape change in CodeUpdated between ledger v8 and v9 causes the decode to fail, making the event appear absent even on a successful upgrade.

The bug is confirmed present in main - federatedRuntimeUpgrade.ts is unchanged since before this issue was filed and still has the hasEvent check at line 63.

PR #1973 (wes-1960-runtime-upgrade-success-check) addresses Consequence 1 directly: it replaces the event-check with a spec_version comparison via state_getRuntimeVersion, which is immune to the metadata mismatch. The PR is mergeable but is currently behind main.

Consequence 2 (chain-indexer exiting at the upgrade block with Can't decode event topics) is not covered by PR #1973 and remains open - it sits in the indexer-common/subxt client layer and needs a separate owner.

Next actions: merge PR #1973 to close Consequence 1; raise a separate issue (or assign this one's Consequence 2) against chain-indexer/indexer-common to fix the event decode failure at the upgrade block.

---

Claude triage review - an automated pass over open bugs with little recent activity. Conclusions come from reading this issue plus the current code and git history, and may be wrong; please correct anything that is off.