feat: statistical range key-count estimation (getEstimatedKeyCount range support + CountEstimator)

README.md

@@ -432,13 +432,75 @@ the `expectedVersion` option is used.
 ### `db.getEstimatedKeyCount(): number`
 
 Retrieves the estimated number of keys in the database. This is an alias for
-`db.getDBIntProperty('rocksdb.estimate-num-keys')`.
+`db.getDBIntProperty('rocksdb.estimate-num-keys')`; use `estimateCount()` for range support and a
+confidence indicator.
 
 ```typescript
 const estimated = db.getEstimatedKeyCount();
 console.log(estimated);
 ```
 
+### `db.estimateCount(options?: CountEstimateOptions): CountEstimate`
+
+Estimates the number of keys in the database, or within a key range, returning
+`{ count, confidence }`. Unlike `getKeysCount()`, this never iterates: the estimate is derived
+from RocksDB statistics (memtable stats plus approximate SST sizes converted through the entry
+density of the SSTs overlapping the range), so its cost scales with the number of SSTs overlapping
+the range rather than the number of keys. Reading cold table properties can do I/O through the
+table cache, so narrow ranges are preferable. A start-only range is computed as the
+whole-database estimate minus the complement, so it does the work of the range _below_ `start`.
+Accuracy improves with range size. Resolution is bounded by SST data-block granularity, so a range
+narrower than a block is unreliable in either direction: it may over-report or report 0 for present
+keys, and its low `confidence` is the signal. Recently deleted or overwritten entries may be counted
+until compaction.
+Estimates always reflect committed state; writes pending in a transaction are not included. Set
+`reverse: true` to use `getRange()`'s reverse convention (`start` is the upper bound and `end` is
+the lower bound). An inverted range (`start` ≥ `end`) returns
+`{ count: 0, confidence: 1 }`.
+
+`confidence` is a heuristic 0–1 indicator of how trustworthy `count` is — exactly 1 only when the
+count is exact. It is derived from the estimate's resolution (data-block/memtable-sampling
+granularity relative to the count), the tombstone fraction of the overlapping SSTs, and — for
+start-only ranges — the error compounded by complement subtraction. Treat it as an ordering
+signal (e.g. when to trust an estimate for query planning vs fall back to a heuristic), not a
+statistical bound.
+
+```typescript
+const { count, confidence } = db.estimateCount({ start: 'a', end: 'z' });
+```
+
+### `db.createCountEstimator(options?: CountEstimatorOptions): CountEstimator`
+
+Creates an estimator that progressively refines a range count estimate while the range is being
+iterated — useful for reporting a total alongside a page of results without scanning the full
+range. Before any traversal, `estimate()` returns the pure statistical estimate (same as
+`estimateCount(range)`). As the caller reports progress with `advance(lastKey, count)` (e.g. once
+per page), `estimate()` returns the exact traversed count plus a statistical estimate of the
+remainder, calibrated by the observed ratio of actual-to-estimated entries over the portion
+already traversed — so the count converges toward the exact total. `confidence` is the
+exactness-weighted blend of the traversed portion and the remainder's confidence, so it approaches 1
+as the exact portion grows, although a checkpoint may decrease when calibration makes a large
+correction.
+Each checkpoint reads committed state, so a traversal performed against a transaction snapshot may
+be calibrated against data committed after that snapshot.
+When traversal completes, call `finish()` and `estimate()` returns the exact count with
+confidence 1. Reverse ranges follow `getRange`: set `start` to the upper bound and `end` to the
+lower bound, then set `reverse: true`. The caller owns the progress contract: cursors must move
+monotonically through the range and each entry must be reported exactly once.
+
+```typescript
+const range = { start: 'a', end: 'z' };
+const estimator = db.createCountEstimator(range);
+let lastKey;
+let pageSize = 0;
+for (const { key } of db.getRange({ ...range, limit: 25 })) {
+	lastKey = key;
+	pageSize++;
+}
+estimator.advance(lastKey, pageSize);
+const { count, confidence } = estimator.estimate();
+```
+
 ### `db.getKeys(options?: IteratorOptions): ExtendedIterable`
 
 Retrieves all keys within a range.

src/binding/database/database.cpp

@@ -922,6 +922,205 @@ napi_value Database::GetCount(napi_env env, napi_callback_info info) {
 	return result;
 }
 
+struct RangeEstimate {
+	double count = 0;
+	double memtableCount = 0;
+	double sstCount = 0;
+	double entriesPerBlock = 0;
+	double liveFraction = 1;
+	bool degraded = false;
+};
+
+/**
+ * Estimates the number of live keys in `[start, end)` from RocksDB statistics
+ * alone — no iteration:
+ *
+ * - memtable portion: `GetApproximateMemTableStats` returns an entry count
+ *   directly (it counts all memtable entries, including tombstones and
+ *   overwrites, so it can over-report a recently-deleted range).
+ * - SST portion: the approximate file bytes covered by the range
+ *   (`GetApproximateSizes`) converted to entries using the live-entry density
+ *   of only the SSTs overlapping the range (`GetPropertiesOfTablesInRange`:
+ *   `(num_entries - num_deletions) / file bytes`). Using range-local table
+ *   properties keeps the density honest when entry sizes vary across the
+ *   keyspace, and needs no cache/invalidation.
+ *
+ * Overlapping versions of a key in multiple levels are counted once per
+ * level, so the estimate skews high on heavily-overwritten ranges until
+ * compaction; resolution is bounded by SST data-block granularity, so tiny
+ * ranges can over-report or report zero for present keys.
+ */
+static RangeEstimate estimateRangeCount(rocksdb::DB* db, rocksdb::ColumnFamilyHandle* cf, const rocksdb::Slice& start, const rocksdb::Slice& end) {
+	rocksdb::Range range(start, end);
+	RangeEstimate result;
+
+	uint64_t memtableCount = 0;
+	uint64_t memtableSize = 0;
+	db->GetApproximateMemTableStats(cf, range, &memtableCount, &memtableSize);
+	result.memtableCount = static_cast<double>(memtableCount);
+	result.count = result.memtableCount;
+
+	rocksdb::SizeApproximationOptions sizeOptions;
+	sizeOptions.include_memtables = false;
+	sizeOptions.files_size_error_margin = 0.1;
+	uint64_t sstBytes = 0;
+	rocksdb::Status status = db->GetApproximateSizes(sizeOptions, cf, &range, 1, &sstBytes);
+	if (!status.ok()) {
+		result.degraded = true;
+		return result;
+	}
+
+	rocksdb::TablePropertiesCollection props;
+	status = db->GetPropertiesOfTablesInRange(cf, &range, 1, &props);
+	uint64_t entries = 0;
+	uint64_t deletions = 0;
+	uint64_t fileBytes = 0;
+	uint64_t dataBlocks = 0;
+	if (status.ok()) {
+		for (const auto& prop : props) {
+			if (!prop.second) {
+				result.degraded = true;
+				continue;
+			}
+			const rocksdb::TableProperties& p = *prop.second;
+			entries += p.num_entries;
+			deletions += p.num_deletions;
+			// Approximate the on-disk file size covered by table properties;
+			// GetApproximateSizes offsets span data + index + filter blocks,
+			// so the density denominator must too.
+			fileBytes += p.data_size + p.index_size + p.filter_size;
+			dataBlocks += p.num_data_blocks;
+		}
+	} else {
+		result.degraded = true;
+	}
+	if (entries <= deletions || fileBytes == 0) {
+		// A nonzero byte estimate without density leaves the SST portion unknown.
+		result.degraded = result.degraded || sstBytes != 0;
+		return result;
+	}
+	if (sstBytes == 0) {
+		result.degraded = true;
+		return result;
+	}
+
+	double density = static_cast<double>(entries - deletions) / static_cast<double>(fileBytes);
+	result.sstCount = static_cast<double>(sstBytes) * density;
+	result.count += result.sstCount;
+	result.entriesPerBlock = dataBlocks > 0
+		? static_cast<double>(entries - deletions) / static_cast<double>(dataBlocks)
+		: 0;
+	result.liveFraction = static_cast<double>(entries - deletions) / static_cast<double>(entries);
+	return result;
+}
+
+/**
+ * Heuristic [0, 1] trust indicator for a range estimate — 1 only when exact.
+ * Combines the estimate's resolution (SST portion is quantized to data-block
+ * granularity, memtable counts to skip-list sampling granularity) with the
+ * tombstone fraction of the overlapping SSTs (a proxy for overwrite/delete
+ * skew the estimate cannot see).
+ */
+static double estimateConfidence(const RangeEstimate& est) {
+	if (est.degraded) {
+		return 0.1;
+	}
+	if (est.count <= 0) {
+		return 0.95;
+	}
+	double sstResolution = std::max(est.entriesPerBlock, 1.0);
+	double memtableResolution = 8;
+	double resolution = (est.sstCount * sstResolution + est.memtableCount * memtableResolution) / est.count;
+	double granularity = est.count / (est.count + resolution);
+	return granularity * (0.5 + 0.5 * est.liveFraction);
+}
+
+/**
+ * Estimates the number of keys within a range without iterating, returning
+ * `{ count, confidence }`. Both keys are optional buffers; an open-ended side
+ * is handled by subtracting the complementary range from the
+ * whole-column-family `estimate-num-keys` (an empty slice is the *smallest*
+ * key, so it must never be passed as an upper bound).
+ *
+ * @example
+ * ```typescript
+ * const db = NativeDatabase.open('path/to/db');
+ * const { count, confidence } = db.estimateCount(startBuffer, endBuffer);
+ * ```
+ */
+napi_value Database::EstimateCount(napi_env env, napi_callback_info info) {
+	NAPI_METHOD_ARGV(2);
+	UNWRAP_DB_HANDLE_AND_OPEN();
+	ACQUIRE_OPERATIONS_LOCK();
+
+	rocksdb::DB* db = (*dbHandle)->descriptor->db.get();
+	rocksdb::ColumnFamilyHandle* cf = (*dbHandle)->getColumnFamilyHandle();
+
+	// N-API may return a null data pointer for a zero-length buffer.
+	void* startData = nullptr;
+	size_t startLength = 0;
+	napi_valuetype startType;
+	NAPI_STATUS_THROWS(::napi_typeof(env, argv[0], &startType));
+	bool hasStart = startType != napi_undefined && startType != napi_null;
+	if (hasStart) {
+		NAPI_STATUS_THROWS(::napi_get_buffer_info(env, argv[0], &startData, &startLength));
+	}
+
+	void* endData = nullptr;
+	size_t endLength = 0;
+	napi_valuetype endType;
+	NAPI_STATUS_THROWS(::napi_typeof(env, argv[1], &endType));
+	bool hasEnd = endType != napi_undefined && endType != napi_null;
+	if (hasEnd) {
+		NAPI_STATUS_THROWS(::napi_get_buffer_info(env, argv[1], &endData, &endLength));
+	}
+
+	rocksdb::Slice startSlice(startLength ? static_cast<const char*>(startData) : "", startLength);
+	rocksdb::Slice endSlice(endLength ? static_cast<const char*>(endData) : "", endLength);
+
+	double estimate = 0;
+	double confidence = 0;
+	if (!hasEnd) {
+		uint64_t totalKeys = 0;
+		bool totalOk = db->GetIntProperty(cf, rocksdb::DB::Properties::kEstimateNumKeys, &totalKeys);
+		double total = static_cast<double>(totalKeys);
+		if (!totalOk) {
+			estimate = 0;
+			confidence = 0;
+		} else if (!hasStart || startLength == 0 || totalKeys == 0) {
+			estimate = total;
+			confidence = totalKeys == 0 ? 0.95 : 0.9;
+		} else {
+			// No upper bound: estimate [start, ∞) as total minus [min, start).
+			RangeEstimate complement = estimateRangeCount(db, cf, rocksdb::Slice(), startSlice);
+			estimate = std::max(0.0, total - complement.count);
+			double share = estimate / std::max(estimate + complement.count, 1.0);
+			confidence = std::min(0.9, estimateConfidence(complement)) * share;
+		}
+	} else if (endLength == 0 || startSlice.compare(endSlice) >= 0) {
+		// Empty end bound (below every key) or inverted/empty range:
+		// GetApproximateSizes would underflow (end offset minus start offset
+		// in uint64). Comparator is always bytewise (db_descriptor.cpp), so
+		// Slice::compare matches key order. Empty by construction, so exact.
+		estimate = 0;
+		confidence = 1.0;
+	} else {
+		RangeEstimate rangeEstimate = estimateRangeCount(db, cf, startSlice, endSlice);
+		estimate = rangeEstimate.count;
+		confidence = estimateConfidence(rangeEstimate);
+	}
+
+	napi_value result;
+	NAPI_STATUS_THROWS(::napi_create_object(env, &result));
+	napi_value countValue;
+	NAPI_STATUS_THROWS(::napi_create_double(env, std::round(estimate), &countValue));
+	NAPI_STATUS_THROWS(::napi_set_named_property(env, result, "count", countValue));
+	napi_value confidenceValue;
+	NAPI_STATUS_THROWS(::napi_create_double(env, confidence, &confidenceValue));
+	NAPI_STATUS_THROWS(::napi_set_named_property(env, result, "confidence", confidenceValue));
+	return result;
+}
+
 napi_value Database::GetMonotonicTimestamp(napi_env env, napi_callback_info info) {
 	NAPI_METHOD();
 	UNWRAP_DB_HANDLE_AND_OPEN();
@@ -1987,6 +2186,7 @@ void Database::Init(napi_env env, napi_value exports) {
 		{ "destroy", nullptr, Destroy, nullptr, nullptr, nullptr, napi_default, nullptr },
 		{ "drop", nullptr, Drop, nullptr, nullptr, nullptr, napi_default, nullptr },
 		{ "dropSync", nullptr, DropSync, nullptr, nullptr, nullptr, napi_default, nullptr },
+		{ "estimateCount", nullptr, EstimateCount, nullptr, nullptr, nullptr, napi_default, nullptr },
 		{ "flush", nullptr, Flush, nullptr, nullptr, nullptr, napi_default, nullptr },
 		{ "flushSync", nullptr, FlushSync, nullptr, nullptr, nullptr, napi_default, nullptr },
 		{ "get", nullptr, Get, nullptr, nullptr, nullptr, napi_default, nullptr },

src/binding/database/database.h

@@ -285,6 +285,7 @@ struct Database final {
 	static napi_value FlushSync(napi_env env, napi_callback_info info);
 	static napi_value Get(napi_env env, napi_callback_info info);
 	static napi_value GetCompression(napi_env env, napi_callback_info info);
+	static napi_value EstimateCount(napi_env env, napi_callback_info info);
 	static napi_value GetCount(napi_env env, napi_callback_info info);
 	static napi_value GetDBIntProperty(napi_env env, napi_callback_info info);
 	static napi_value GetDBProperty(napi_env env, napi_callback_info info);