perf(storage): prototype compact PieceV2 storage

[Kubuxu]

Summary

Draft implementation of the compact two-slot PieceV2 storage prototype, related to #286.

New compact-state datasets store pieces in a contiguous per-dataset array:

struct PieceV2 {
    bytes32 root;
    uint256 metadata;
}

metadata packs padding, tree height, leaf count, and the Fenwick partial sum into one storage slot.

Changes

• Hardened canonical PieceCIDv2 validation and decoded (padding, height, root) in one pass.
• Added compact PieceV2 storage while retaining the existing declarations for physical layout safety.
• Moved piece addition, proof verification, lookup, getters, pagination, CID search, scheduling, removal, and cleanup to compact storage.
• Preserved piece IDs, public ABI behavior, events, and listener callback arguments for compact-state datasets.
• Added coverage for packing bounds, Fenwick semantics, proofs, pagination, removal, cleanup, and raw storage reclamation.
• Measured steady-state addPieces storage activity for 1, 4, 16, and 32-piece batches.

Storage measurements

Each scenario creates a fresh dataset, adds one seed piece before recording, then records the measured addPieces call.

Batch	EVM-slot reads	EVM-slot writes	KAMT touched	KAMT modified	Newly occupied slots
1	10 → 12	7 → 4	11 → 7	6 → 3	5 → 2
4	21 → 25	28 → 13	24 → 7	19 → 3	20 → 8
16	69 → 85	112 → 49	72 → 8	67 → 4	80 → 32
32	133 → 165	224 → 97	137 → 9	132 → 5	160 → 64

Values are legacy → compact.

Findings

• Storage footprint: consistently 60% lower—5N → 2N newly occupied slots.
• EVM-slot writes: reduced from 7N to 3N + 1; reduction grows from 42.9% at one piece to 56.7% at 32 pieces.
• KAMT locality: compact contiguous records substantially reduce object activity as batches grow:
  • 4 pieces: 24 → 7 touched, 19 → 3 modified.
  • 16 pieces: 72 → 8 touched, 67 → 4 modified.
  • 32 pieces: 137 → 9 touched, 132 → 5 modified.
• EVM-slot reads: increase by 19.0–24.1%. These are repeated reads of already-accessed values, which are cached; they do not correspond to a comparable increase in KAMT object activity.
• Scaling: the compact layout modifies two shared objects plus roughly one compact-data object per 16 pieces, subject to alignment. The legacy representation modifies approximately four piece-data objects per piece.

At 32 pieces, the compact layout occupies 64 slots instead of 160, touches 9 KAMT objects instead of 137, and modifies 5 instead of 132.

Compatibility note

This is a forward-only prototype. Backwards compatibility for datasets created before the compact representation was not considered.

Keep this PR as draft; it is not ready to merge or deploy as an upgrade.

Verification

make test
209 passed, 0 failed

[Kubuxu]

Benchmark results:

Optimised gas reduction versus baseline:

Ballast slots	Batch 1	2	4	8	16	32
0	9.9%	9.1%	27.5%	33.1%	36.2%	38.1%
1M	40.4%	57.4%	72.8%	82.3%	87.0%	89.5%
5M	40.3%	60.7%	73.6%	83.5%	88.3%	91.1%

Flat gas used

Ballast slots	Version	Batch 1	2	4	8	16	32
0	Baseline	17,217,928	22,999,322	42,969,779	64,817,453	112,656,392	205,053,901
0	Optimized	15,504,996	20,909,427	31,155,721	43,346,504	71,885,266	126,894,938
1M	Baseline	135,895,549	200,896,826	341,493,403	592,579,677	989,314,920	1,817,386,096
1M	Optimized	80,992,902	85,593,374	92,775,205	105,070,523	128,651,224	190,157,808
5M	Baseline	157,548,595	249,842,564	389,163,815	711,535,675	1,251,005,462	2,259,544,725
5M	Optimized	94,115,959	98,222,892	102,830,002	117,059,589	146,507,177	200,297,037

The optimisation largely removes ballast sensitivity: at 5M slots, batch 32 falls from 2.260B to 200.3M gas (91.1%, or 11.3× lower). Batch 64 is excluded because both versions revert at the Lotus event-size limit.

Benchmarks were performed in foc-devnet, ballasting down the PDPVerifier tree with the added dev-only balast(start, n) method creating sputtering of occupied slots.

[Kubuxu]

I plan to remove this doc. It was generated as part of LLM task guidance.

[Kubuxu]

@rvagg @wjmelements I would appreciate an early review. I don't expect the logic to change much, but it will get way messier when I make this backwards compatible.

[rvagg]

Batch 64 is excluded because both versions revert at the Lotus event-size limit.

But I think we can fix that here can't we because (a) we're emitting those events with uncapped arrays of piece CIDs and (b) the piece CID we're emitting are the larger legacy format, not the compact 64-byte format; so we could go quite high, at least for the PDPVerifier component (FWSS will impose its own limits), right?