Add AdaptiveRadixTree string-property index (ART + SIMD + leaf fingerprint) - #699
Open
rjb32 wants to merge 10 commits into
Open
Add AdaptiveRadixTree string-property index (ART + SIMD + leaf fingerprint)#699rjb32 wants to merge 10 commits into
rjb32 wants to merge 10 commits into
Conversation
docs/ART.md specifies the chosen read path (ART + SIMD Node16 + leaf fingerprint miss-reject + full-leaf verify + AMAC 8-way batched probing). tools/index-sim/ holds the prototypes and measurement reports it cites (compiled binaries excluded).
…print) Templated, string_view-keyed in-memory ART implementing the docs/ART.md read path: adaptive Node4/16/48/256 with path compression and lazy expansion, SIMD child search, a 16-bit leaf-key fingerprint carried in the leaf child pointer for miss-reject, full-key verify at the leaf, and an 8-way AMAC-pipelined batched lookup. The node machinery is type-erased into the .cpp behind a thin templated facade.
…hmark Wire google/benchmark v1.9.5 as an external submodule (built on demand) and add test/storage/AdaptiveRadixTreeBenchmark.cpp comparing the ART against a transparent-hash std::unordered_map on build, serial hit, prefix-colliding miss, and batched-probe throughput over 1K/128K/1M 16-byte keys.
rjb32
changed the title
Reproduce the tools/index-sim multiversion hash (open-addressed, FNV-1a per probe, version-chain slots) and benchmark it alongside the ART and std::unordered_map on build, hit, prefix-colliding miss, and batched probing -- the ART-vs-hash comparison docs/ART.md is built around.
Every node and leaf now comes from a per-tree bump arena instead of an individual new, co-locating a tree's memory to cut the cache/TLB misses that dominate the descent. Build drops ~5.7x at 1M keys (no per-node malloc/free), batched probes overtake the multiversion hash, and serial hits improve in the cache-resident range. The deep-DRAM 1M serial hit is unchanged: it is bound by dependent-miss count, which only height reduction addresses. Node growth leaves the outgrown node as dead arena space (bounded; a node grows at most three times); the whole arena is freed when the tree is destroyed.
…card Document that multi-byte stride / HOT-family height reduction was prototyped against the production arena-backed ART to close the deep-DRAM serial point read, and rejected: the only stride that helps (S=4) collapses the upper tree into a prefix-hash, forfeiting the ordered/range scans and adaptive sizing the ART exists for. Note the arena was kept. Adds a docs/ART.md rejected-alternative bullet, a production follow-up to report_art_opt.md, and a scope note to report_art.md (its hash-front + radix-tail hybrid is distinct and untested).
…h MVCC hash Measures 'read a key as of commit C' over a 12-commit timeline with moderate per-commit churn, using the real db::CommitHash and TuringDB's visibility rule (index(V) <= index(C) via the offsets map; committed history is linear because branches rebase in at submit). The ART uses a retained root per commit (one flat traversal per read); the MVCC hash walks a CommitHash-tagged per-key version chain. ART read latency is flat in lag; the hash degrades as the chain walk deepens with commit age. Cross-checked (hash as-of == ART per commit) and ASan/UBSan-clean.
Move the index-sim cost-model study from tools/ into samples/index-sim/ and wire it into the samples build: index-sim (the 0.025s cost model) is the CI sample via turing_sample; index-bench builds as a plain executable (it times real structures for ~20s, validating the model, so it stays out of test.list). Remove index_decompose/index_miss/index_opt and their report_art*.md reports as dead code, keeping only index_sim/index_bench and their reports.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implement an Adaptive Radix Tree string index based on simulations and benchmarks against a multiversion hash table. The ART uses SIMD for key search in nodes and CRC fingerprinting to accelerate the miss case against the hash table.
The MVCC hash table currently has only a uint32_t version ID for linear version comparison, does not support branches.
Throughput at 1M high-cardinality 16-byte keys (M ops/s, higher is better):
Read checked out at a commit over a 12-commit timeline, with a CommitHash-tagged MVCC hash (M reads/s, higher is better):