feat: statistical range key-count estimation (getEstimatedKeyCount range support + CountEstimator) - #778
Merged
Merged
Conversation
Adds Database::EstimateCount — a no-iteration range key-count estimate built from RocksDB statistics: GetApproximateMemTableStats supplies the memtable entry count directly, and the SST portion converts approximate file bytes in range (GetApproximateSizes) to entries via the live-entry density of only the SSTs overlapping the range (GetPropertiesOfTablesInRange: (num_entries - num_deletions) / file bytes). Open-ended ranges subtract the complementary range from estimate-num-keys rather than passing an empty upper-bound slice (which would denote the smallest key). Public API: getEstimatedKeyCount(options?: RangeOptions) extends the existing whole-DB method with range support, and createCountEstimator() returns a CountEstimator that rides an iterator: advance(lastKey, n) checkpoints progress and estimate() returns the exact traversed count plus a remainder estimate calibrated by the observed actual/estimated ratio over the traversed portion, converging toward the exact total. Closes #205 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- guard inverted/empty bounded ranges (GetApproximateSizes would underflow end-start offsets in uint64) — returns 0 - honor exclusiveStart/inclusiveEnd by appending the bytewise-successor zero byte to the encoded bound - CountEstimator: exclude the cursor entry from the remainder (forward mode double-counted it, blocking convergence), add finish() as the completion signal, memoize estimate() per checkpoint, and document the caller-owned progress contract - temper the cost claims: scales with overlapping SSTs, table-property reads can do I/O for cold files, start-only ranges do complement work Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
napi can return a null data pointer for a zero-length buffer, which the previous guard read as an omitted bound — an empty end bound (below every key) became a whole-database estimate on the NativeDatabase surface (encodeKey shields the public API). Track presence explicitly: empty end returns 0, empty start is the minimum key. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Code Review
This pull request implements a non-iterating key-count estimation feature for RocksDB ranges, exposing getEstimatedKeyCount with range options and introducing a new CountEstimator class to progressively refine estimates during iteration. The review feedback correctly identifies that CountEstimator currently discards the exclusiveStart and inclusiveEnd options from CountEstimatorOptions, and suggests storing and utilizing these options in the estimate() method to ensure bounds are correctly respected.
Contributor
📊 Benchmark Resultsget-sync.bench.tsgetSync() > random keys - small key size (100 records)
getSync() > sequential keys - small key size (100 records)
ranges.bench.tsgetRange() > small range (100 records, 50 range)
realistic-load.bench.tsRealistic write load with workers > write variable records with transaction log
transaction-log.bench.tsTransaction log > read 100 iterators while write log with 100 byte records
Transaction log > read one entry from random position from log with 1000 100 byte records
worker-put-sync.bench.tsputSync() > random keys - small key size (100 records, 10 workers)
worker-transaction-log.bench.tsTransaction log with workers > write log with 100 byte records
Results from commit ba79781 |
Per review feedback on the API shape: a bare number hides how much an
estimate should be trusted. New db.estimateCount(options?) returns
{ count, confidence }; getEstimatedKeyCount() reverts to its original
no-arg number signature (kept as the cheap estimate-num-keys alias), so
one name no longer covers two cost profiles. CountEstimator.estimate()
returns the same shape.
confidence is a heuristic [0,1], exactly 1 only when the count is exact
(finish(), inverted/empty-by-construction ranges). Computed natively
from the estimate components: resolution (SST data-block / memtable
sampling granularity relative to the count), tombstone fraction of the
overlapping SSTs, and for start-only ranges the error compounded by
complement subtraction. Measured on 500k varied entries: 0.999 on
full/half ranges (~3% error), 0.88 at 1%, 0.21 on a 50-key range (~2x
over-report), 0.13 on a start-only tail (+39% — complement subtraction
correctly distrusted); estimator confidence converges to 1.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adjudicated major from the API-shape review: a failed GetApproximateSizes/
GetPropertiesOfTablesInRange silently degraded to a memtable-only count
while the confidence formula still reported it as trustworthy, and a
failed estimate-num-keys property read returned { 0, 1.0 } — a missing
answer dressed as a confidently empty database. Track degradation in
RangeEstimate (capping confidence at 0.1) and return { 0, 0 } for the
failed property read. Also: guard null table-properties entries, honor
the range own exclusiveStart/inclusiveEnd flags in CountEstimator
segments, and cap non-exact estimator confidence at 0.999 so only
finish() and exact-by-construction ranges claim 1.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…degrade Follow-ups from the delta review: a successful zero estimate-num-keys read now reports 0.95 confidence (deletion entries can offset puts, so even zero is estimated), and a null entry in the table-properties collection marks the density degraded rather than being silently skipped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
kriszyp
marked this pull request as ready for review
August 13, 2026 18:28
cb1kenobi
reviewed
Aug 14, 2026
cb1kenobi
reviewed
Aug 15, 2026
kriszyp
added a commit
to HarperFast/harper
that referenced
this pull request
Aug 15, 2026
Range comparators (starts_with/prefix, between and friends, lt/le/gt/ge) have always estimated as fixed fractions of the table size (5%/10%/30%), which makes condition ordering — and #2147 count=estimated totals — wildly wrong for any real range. When the store provides rocksdb-js estimateCount ({ count, confidence }), estimateCondition now estimates the actual range the search would iterate (mirroring searchByIndex range construction), blended with the old fraction heuristic by the estimate confidence, so low-confidence estimates (block-granular tiny ranges, open-ended complement subtraction) degrade gracefully to the previous behavior. RocksIndexStore translates value-space bounds to its composite [indexedValue, primaryKey] keys ([value, MAXIMUM_KEY]), mirroring its getRange. estimatedEntryCount switches from an exact full-store getKeysCount scan (every 10s per store) to the O(1) estimate-num-keys read. The capability is feature-detected (typeof store.estimateCount), so behavior is unchanged until the rocksdb-js dependency ships HarperFast/rocksdb-js#778; the end-to-end tests self-skip until then.
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
cb1kenobi
reviewed
Aug 15, 2026
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
cb1kenobi
reviewed
Aug 17, 2026
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Member
|
Reviewed — |
kriszyp
added a commit
to HarperFast/harper
that referenced
this pull request
Aug 26, 2026
Range comparators (starts_with/prefix, between and friends, lt/le/gt/ge) have always estimated as fixed fractions of the table size (5%/10%/30%), which makes condition ordering — and #2147 count=estimated totals — wildly wrong for any real range. When the store provides rocksdb-js estimateCount ({ count, confidence }), estimateCondition now estimates the actual range the search would iterate (mirroring searchByIndex range construction), blended with the old fraction heuristic by the estimate confidence, so low-confidence estimates (block-granular tiny ranges, open-ended complement subtraction) degrade gracefully to the previous behavior. RocksIndexStore translates value-space bounds to its composite [indexedValue, primaryKey] keys ([value, MAXIMUM_KEY]), mirroring its getRange. estimatedEntryCount switches from an exact full-store getKeysCount scan (every 10s per store) to the O(1) estimate-num-keys read. The capability is feature-detected (typeof store.estimateCount), so behavior is unchanged until the rocksdb-js dependency ships HarperFast/rocksdb-js#778; the end-to-end tests self-skip until then.
kriszyp
added a commit
to HarperFast/harper
that referenced
this pull request
Aug 27, 2026
Range comparators (starts_with/prefix, between and friends, lt/le/gt/ge) have always estimated as fixed fractions of the table size (5%/10%/30%), which makes condition ordering — and #2147 count=estimated totals — wildly wrong for any real range. When the store provides rocksdb-js estimateCount ({ count, confidence }), estimateCondition now estimates the actual range the search would iterate (mirroring searchByIndex range construction), blended with the old fraction heuristic by the estimate confidence, so low-confidence estimates (block-granular tiny ranges, open-ended complement subtraction) degrade gracefully to the previous behavior. RocksIndexStore translates value-space bounds to its composite [indexedValue, primaryKey] keys ([value, MAXIMUM_KEY]), mirroring its getRange. estimatedEntryCount switches from an exact full-store getKeysCount scan (every 10s per store) to the O(1) estimate-num-keys read. The capability is feature-detected (typeof store.estimateCount), so behavior is unchanged until the rocksdb-js dependency ships HarperFast/rocksdb-js#778; the end-to-end tests self-skip until then.
kriszyp
added a commit
to HarperFast/harper
that referenced
this pull request
Aug 27, 2026
Range comparators (starts_with/prefix, between and friends, lt/le/gt/ge) have always estimated as fixed fractions of the table size (5%/10%/30%), which makes condition ordering — and #2147 count=estimated totals — wildly wrong for any real range. When the store provides rocksdb-js estimateCount ({ count, confidence }), estimateCondition now estimates the actual range the search would iterate (mirroring searchByIndex range construction), blended with the old fraction heuristic by the estimate confidence, so low-confidence estimates (block-granular tiny ranges, open-ended complement subtraction) degrade gracefully to the previous behavior. RocksIndexStore translates value-space bounds to its composite [indexedValue, primaryKey] keys ([value, MAXIMUM_KEY]), mirroring its getRange. estimatedEntryCount switches from an exact full-store getKeysCount scan (every 10s per store) to the O(1) estimate-num-keys read. The capability is feature-detected (typeof store.estimateCount), so behavior is unchanged until the rocksdb-js dependency ships HarperFast/rocksdb-js#778; the end-to-end tests self-skip until then.
kriszyp
added a commit
to HarperFast/harper
that referenced
this pull request
Aug 31, 2026
…2163) * Use storage-level statistical range estimates in the query planner Range comparators (starts_with/prefix, between and friends, lt/le/gt/ge) have always estimated as fixed fractions of the table size (5%/10%/30%), which makes condition ordering — and #2147 count=estimated totals — wildly wrong for any real range. When the store provides rocksdb-js estimateCount ({ count, confidence }), estimateCondition now estimates the actual range the search would iterate (mirroring searchByIndex range construction), blended with the old fraction heuristic by the estimate confidence, so low-confidence estimates (block-granular tiny ranges, open-ended complement subtraction) degrade gracefully to the previous behavior. RocksIndexStore translates value-space bounds to its composite [indexedValue, primaryKey] keys ([value, MAXIMUM_KEY]), mirroring its getRange. estimatedEntryCount switches from an exact full-store getKeysCount scan (every 10s per store) to the O(1) estimate-num-keys read. The capability is feature-detected (typeof store.estimateCount), so behavior is unchanged until the rocksdb-js dependency ships HarperFast/rocksdb-js#778; the end-to-end tests self-skip until then. * Address cross-model review: negation inversion, dependency-shape guards - negated conditions now estimate the complement of their positive estimate (root fix in estimateConditionForTable — also covers the pre-existing negated-equals defect): a narrow negated range previously looked highly selective, won the condition ordering, and executed as a full scan - the estimate path validates {count, confidence} shape and wraps the native call in try/catch, so a future dependency bump (or a concurrently closing store) degrades to the fraction heuristic instead of NaN-poisoning plan ordering or failing the request - over-length string bounds fall back (execution truncates at MAX_SEARCH_KEY_LENGTH + filters, so the executed range is wider than the estimable one) - intersectionEstimate divisor floored at 1 - comment trims per review * Document query-plan range estimation invariants in DESIGN.md * Negated conditions estimate Infinity, matching the full-scan convention The complement inversion still let a wide positive range make its negation look cheap while executing as a full scan. estimated_count feeds driving-condition ordering, and the codebase already encodes full-scan cost as Infinity for the filter-only comparators (contains/ends_with) — negated conditions (which always force needFullScan) now follow the same convention. * Bump rocksdb-js to 2.8.0 and use the shipped estimateCount API rocksdb-js 2.8.0 ships the statistical range estimation the query planner was written against, so the RocksIndexStore override becomes a real typed method mirroring getRange (reverse flip included) instead of a conditional prototype assignment, and the real-store estimate tests run unconditionally. The per-store capability probe and result-shape validation stay: LMDB-backed and custom index stores don't implement estimateCount. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Gate the real-store estimate suite on the live store, not the Rocks prototype test:unit:lmdb re-runs test:unit:resources with HARPER_STORAGE_ENGINE=lmdb, where the index stores have no estimateCount at all. Keying the skip off RocksDatabase.prototype was already wrong and became always-true once 2.8.0 was pinned, so gate on the table's actual index store instead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Estimate lt/le over the range execution actually iterates searchByIndex bounds lt/le at `start: true`, which sorts above `null`, so an indexNulls index's `[null, primaryKey]` entries are outside the executed range. estimateRangeCondition left the lower bound open and counted them: on an index that is 99% nulls the estimate came back 21x high (19960 for a 200-row condition), so a genuinely selective condition looked like a full scan and lost the driving-condition ordering — worse than the flat heuristic it replaces. getRange and estimateCount now share one translateIndexBounds helper so the "estimate the range you execute" invariant is structural instead of two copies, and a spy test pins that both forward identical bounds, reverse branch included. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Pin estimates against executed result counts, not just against each other The real-store suite compared estimates to one another (narrow < wide), which passes even if every estimate is an order of magnitude off. Three cases now run the actual query and assert the estimate lands within 10x of what it returned, covering the primary-key path as well — that path passes `start: true` through to the primary store, and nothing pinned it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Align root msgpackr pin with rocksdb-js 2.8.0's dependency rocksdb-js 2.8.0 depends on msgpackr@2.0.6; the root package.json still pinned 2.0.5, so npm couldn't dedupe to a single msgpackr instance and the shrinkwrap-pin smoke check failed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Fix DESIGN.md formatting after rebase conflict resolution Prettier wants blank lines around the two independently-appended sections merged during the rebase onto main. * Short-circuit negated range conditions before the native estimate call condition.negated forced estimated_count to Infinity at the end of estimateConditionForTable, but every branch above it (including the native estimateCount FFI call for range comparators) still ran first and had its result discarded. Hoist the negated check to skip that work entirely — same outcome, no wasted native call. Found by the pre-push independent review (codex, graded leg) on this rebase. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Reject negative counts in the estimateCount result-shape validation estimateRangeCondition checked Number.isFinite(count) and the confidence bound but not count's sign, so a store returning a negative count (a malformed/buggy native response) would pass validation and blend a negative number into the estimate. Found by the pre-push independent review (codex, graded leg) on this rebase. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Reject non-numeric confidence in the estimateCount result-shape validation The malformed-shape guard rejected NaN/negative count but let a non-numeric-but-comparable confidence (e.g. NaN via >= coercion quirks, or an out-of-range non-finite value) slip through the >=/<= comparison. Mirror the count check: require Number.isFinite before bounds-checking. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a statistical range key-count estimate that never iterates, for query planning and pagination reporting (HarperFast/harper#2147). Closes #205, and supersedes the earlier attempt in #311.
Native
estimateCount(start?, end?)combines three RocksDB primitives:GetApproximateMemTableStats— returns an entry count for the memtable portion directly.GetApproximateSizes(files only, 10% error margin) — approximate on-disk bytes covered by the range.GetPropertiesOfTablesInRange— per-SSTnum_entries/num_deletions/ block sizes for only the SSTs overlapping the range, giving a range-local live-entry density ((entries − deletions) / file bytes) that converts bytes → count.Using range-local table properties (instead of #311's whole-CF cached mean with write-count invalidation) keeps the density honest when entry sizes vary across the keyspace and needs no cache or invalidation hooks. Open-ended ranges are estimated as
estimate-num-keysminus the complementary range — an empty slice is the smallest key, so it must never be passed as an upper bound (a correctness bug in #311's open-ended path). Inverted, empty, and zero-length bounds are guarded (they would otherwise underflow RocksDB's uint64 offset subtraction).Public API (shape chosen by Kris: estimates carry a trust signal):
db.estimateCount(options?: RangeOptions)→{ count, confidence }(CountEstimate).confidenceis a heuristic 0–1 trust indicator, exactly 1 only when the count is exact — derived natively from the estimate's resolution (data-block/memtable-sampling granularity relative to the count), the tombstone fraction of overlapping SSTs, complement-subtraction error for start-only ranges, and failed statistics calls (a degraded estimate caps at 0.1; a failedestimate-num-keysread returns{0, 0}, not a confident empty).exclusiveStart/inclusiveEndare honored via the bytewise-successor zero byte.db.getEstimatedKeyCount()— unchanged original no-arg signature (cheapestimate-num-keysalias), so existing callers are untouched and one name doesn't cover two cost profiles.db.createCountEstimator(options?)→CountEstimator— rides an iterator:advance(lastKey, count)checkpoints progress (e.g. once per page),estimate()returns{count, confidence}= exact traversed + remainder calibrated by the observed actual/estimated ratio (clamped 8×), memoized per checkpoint; confidence is the exactness-weighted blend, capped at 0.999 untilfinish()declares the traversal complete (then exact with confidence 1). Supportsreverseand the range bound flags.For the human reviewer
confidenceis now API surface: callers will encode thresholds against it, so retuning the formula changes their behavior (review ledger point). The semantics doc deliberately promises only "heuristic ordering signal, 1 = exact" — the formula itself is not contract.estimateCount()no-bounds usesestimate-num-keyswhile a bounded full range uses bytes×density; they can disagree. Deliberate — the no-bound path stays O(1) and matchesgetEstimatedKeyCount().CALIBRATION_MIN_TRAVERSED = 16, 8× clamp) are judgment calls; options can be added later.advance()trusts the caller (monotonic cursors, no double-reporting); a wrapping-iterator variant would be additive.Verification
test/estimate-count.test.ts(12 tests): flushed / memtable-only / mixed ranges, open-ended both sides, empty DB (confident 0), inverted range (exact 0), zero-length native bounds, uncommitted-transaction exclusion, monotonic scaling with range width, estimator refinement (confidence must increase), reverse iteration,finish()exactness, and a paginated loop driven to completion (pre-finish confidence < 1, exact total afterfinish()). Full suite green at every commit (latest: 767 passed / 2 skipped, 56 files).[value, primaryKey]keys throughRocksIndexStore), width-ordered estimates confirmed.Review coverage
Generated by Claude (Fable 5). Cross-model pre-push review via
prepush-review.mjs, six rounds:1c4be8d): Codex (graded) + Gemini + Harper-domain adjudication — cursor-composer failed (output-format rejection), cursor-grok pruned. Fixed: inverted-range guard,exclusiveStart/inclusiveEnd, estimator forward off-by-one,finish(), memoization, tempered cost claims.4481bc4): Codex (resumed) + Gemini. Fixed: zero-length end bound bypassing the guard on the native surface (napi nullptr for empty buffers).0ddaf72): Codex + Gemini — hardening confirmed clean.b2cb53a, API change): Codex (graded) + Gemini + Harper-domain — cursor-composer failed again, grok pruned. Adjudicated major fixed in round 5: failed statistics reported as confident estimates.0548d42): Codex (Gemini leg returned no output this round). Follow-ups fixed in round 6: a successful zeroestimate-num-keysclaimed exactness; partial null table-properties collections didn't degrade.89d437b): Codex + Gemini — verdict COMMENTS, both fixes confirmed, no new findings.Accepted, not changed (with rationale): silent-degrade carries low confidence instead of an event; sync-on-JS-thread cost model (documented); caller-owned progress contract; calibration reads current state so concurrent writes behind the cursor can swing a checkpoint (inherent to statistical estimates on a live database, bounded by the 8× clamp); no native GoogleTest for the estimator math (it lives in an N-API translation unit, which cannot link into the gtest target — the vitest suite covers it end-to-end through the real binding).
Human-Review-Need: 3 @ 75562fc