trie-db: Avoid cloning the hashes on the hot path - #234
Closed
lexnv wants to merge 21 commits into
Closed
Conversation
encode_compact emits a detached value node once per referencing trie node, so a value shared by N keys is sent N times (paritytech/polkadot-sdk#12565). Add encode_compact_skip_duplicate_values to emit each distinct value once; repeats are emitted unmodified and stay decodable by existing hash-keyed decoders. For prefixed databases, decode_compact_from_iter_with_known_values re-inserts deduplicated values at every referencing position.
A trailing attached value node was not counted in the returned used-item count, so decoding concatenated encodings at that offset re-read the value bytes as a node.
trie_codec_proof only ran ExtensionLayout (inline values), so value detachment and deduplication were never fuzzed. Round-trip the deduplicating encoding into hash-keyed and prefixed databases, add a hashed-value target with heavily shared values, and a deterministic smoke test so the assertions run in CI without libFuzzer. Also fix the fuzz crate's stale memory-db path dependency version.
An old decoder inserts a deduplicated value once, so its refcount in a hash-keyed database understates the referencing nodes; a consumer consolidating removals could drop a still-referenced value. The re-inserting decoder restores exact parity. Assert full database equality (entries and refcounts, hash-keyed and prefixed) between the plain and deduplicated encodings.
Vendor the compact-proof decoder from the released trie-db 0.31.0 verbatim into the test crate as a frozen snapshot of deployed decoder behavior, and assert that encodings produced by encode_compact_skip_duplicate_values decode with it into a hash-keyed database with every entry readable. This pins the backward compatibility the deduplicating encoder relies on, instead of leaving it as an argument in documentation.
encode_compact_skip_duplicates now emits each distinct trie node once, not just each detached value. A later occurrence of an already-emitted subtree keeps a plain hash reference (like any reference outside the partial trie), so the encoding never grows. The decoder threads a known-items map and re-inserts skipped subtrees at every position, reconstructing the same database as an un-deduplicated encoding.
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Contributor
|
Has been separately integrated in #233 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR addresses the comments from #233 and optimizes the
seen_hashesto take anTrieHashinstead of allocating a vec.