feat(rest): total-count pagination via Prefer: count= (Content-Range)

[cb1kenobi]

Summary

Tracking issue: #2162

Adds opt-in total-record-count for REST collection queries so a client can paginate ("1-25 of 1,234") without a second round-trip or a custom resource.

A GET/HEAD on a collection with a Prefer: count= header gets the total in RFC 7233-style response headers:

GET /Product/?category=software&limit(0,25)
Prefer: count=exact
->
200 OK
Content-Range: items 0-24/1234
Range-Unit: items
Preference-Applied: count=exact

• count=exact — Table.search drains the full matched set once, windowing the requested page in the same pass (O(matched) filter evals, O(limit) memory).
• count=estimated — returns just the page plus a cheap planner/table estimate, no full scan.
• No default — without the header nothing is computed and no headers are emitted.
• Status is always 200 (Content-Range is informational, not 206). HEAD returns the headers with no body — a cheap "how many match?" pre-flight. The three headers are added to Access-Control-Expose-Headers so browsers can read them cross-origin.

count=estimated rides the storage-level range estimator

count=estimated funnels through the query planner's estimateCondition. Since Use storage-level statistical range estimates in the query planner (#2163) merged to main, that planner now derives range-query estimates from rocksdb-js 2.8.0's statistical range estimator (estimateCount) rather than an arbitrary fraction of the table size — so after merging main, this PR's count=estimated reports a range-aware total for between / starts_with / greater-less / open-range queries for free, with no change needed here.

This branch adds only the small piece the count path needs on top of #2163: it exports estimatedEntryCount from search.ts (used by the estimated whole-collection path in Table.ts), and adds REST-count coverage that a range count=estimated produces a valid, range-aware total.

Guardrails and operator control

• An exact count's tail (past the requested page) is bounded by a row cap and a wall-clock budget; on exhaustion the total is reported unavailable (items x-y/*) rather than truncating the page.
• Per-mount config rest: { exactCount: false } serves count=exact as an estimate instead (default enabled) for sensitive/public mounts.

Review findings addressed

Cross-model (Codex) and Harper-domain review surfaced and fixed: read-transaction release on the count path (finally), guardrail no longer truncating the page (+ time budget), estimate corruption by the planner's synthetic sort condition, filter-aware estimates (unknown total instead of a misleading / cardinality-disclosing one), a spurious-header guard for single-record responses, valid-range clamping, and CORS append-not-overwrite. A flagged read-buffer-aliasing concern was verified a non-issue (records return owned copies) and is guarded by a regression test.

Testing

• Resources unit (unitTests/resources/queryCount*.test.js): exact/estimated/window/filtered/default-streaming, plus a Bytes read-buffer-safety guard on both storage engines. Adds primary-key and secondary-index range count=estimated cases proving the total is range-aware (a narrow tail estimates fewer than the whole table/index) via the Use storage-level statistical range estimates in the query planner #2163 estimator.
• REST integration (integrationTests/apiTests/rest.test.mjs): Content-Range/Range-Unit/Preference-Applied, offset window, filtered, estimated, unavailable-total (/*), opt-in, HEAD, and the exactCount gate (its own instance).
• Full resources unit suite green (1882 passing) on the merge with main; tsc and lint clean.

Docs

REST reference docs (Pagination and Total Count, the exactCount option, the Prefer header): HarperFast/documentation#623.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces support for REST pagination total-count using the Prefer: count=exact|estimated header. It updates Table.search to return materialized pages with exact or estimated record counts, incorporating guardrails to prevent unbounded scans. The REST layer is updated to parse the preference, handle configuration-based downgrades, and emit RFC 7233-style headers (Content-Range, Range-Unit, and Preference-Applied). Additionally, new unit and integration tests are added to verify the functionality and ensure read-buffer safety. There are no review comments, and I have no feedback to provide.

[claude]

Reviewed; no blockers found.

[kriszyp]

This is a really cool idea, I like the interface.
However, I agree with the comments; I think we should probably have exactCount disabled by default, it could pretty expensive. I also think this really increases the need for HarperFast/rocksdb-js#311 or some better solution for estimating range counts (maybe should be a prerequisite for this?)
And do we have an issue for this? I'd like to get priorities associated with these (especially if this starts entraining dependency PRs).
_{🤖 Reviewed with Codex}

[cb1kenobi]

Thanks @kriszyp — really helpful review, all five points were spot on. Addressed in 5ec0d38:

Design

• Exact counting is now opt-in (rest: { exactCount: true }, default off); count=exact is otherwise served as an estimate. Estimated stays the safe default, so a public table no longer exposes the exact-scan budget by default.
• Tracking issue: REST pagination total count (Prefer: count=) #2162 — links rocksdb-js#311 as the range-estimation dependency, and flags there the "should better estimation be a prerequisite for broader rollout?" question for prioritization.

Fixes

1. Unbounded/invalid limit — the count path now requires a finite, non-negative integer limit no larger than a max page size (10k); limit(Infinity), limit(foo)→NaN, negative, or oversized limits fall through to streaming with no count.
2. search() contract / DELETE — Prefer: count is now GET/HEAD-only, so a collection DELETE carrying limit()+Prefer keeps the AsyncIterable and deletes instead of receiving a Promise and throwing.
3. Vary: Prefer — emitted on collection reads after serialization (which resets Vary), so shared caches key on Prefer.
4. CORS token comparison — expose-header dedup now compares case-insensitive comma tokens instead of substrings, so X-Content-Range-Metadata no longer suppresses Content-Range.

Docs updated in HarperFast/documentation#623. Tests: unit 11, integration 27 green. Ready for another look when you have a moment.

— Claude Opus 4.8

[kriszyp]

Heads-up for the estimation side of this PR: rocksdb-js #778 — feat: statistical range key-count estimation adds storage-level range estimates that this PR should use once it lands:

• db.estimateCount({ start, end }) → { count, confidence } — a no-iteration statistical range count (memtable stats + range-local SST entry density), measured within ~±5% at 9–83µs where exact scans took 1–85ms. This can replace the planner-fraction path behind count=estimated (estimateCondition's hardcoded 5%/10%/30% guesses for range comparators) with a real per-range number, and confidence gives the REST layer a principled basis for deciding when an estimate is trustworthy enough to report.
• db.createCountEstimator(range) — rides the iteration this PR already does: advance(lastKey, n) per page, estimate() = exact-so-far + calibrated remainder, finish() = exact. This composes nicely with the count=exact guardrail here: when the row-cap/time budget exhausts mid-drain, the partial drain can become a calibrated estimate (with confidence) instead of reporting the total unavailable (items x-y/*).

Harper-side integration is coming as a follow-up PR (planner estimateCondition + estimatedEntryCount, feature-detected so it degrades cleanly on older rocksdb-js). Nothing blocking here — this PR's shape is already right for it.

— Claude (Fable 5)

[kriszyp]

Will post the queued inline comment, but more importantly this probably should switch to use iterator estimator in 778, as noted in the other comment.
_{🤖 Reviewed with Codex}

[kriszyp]

I think we just need to publish a new rocksdb-js version to unblock this so you can switch to the newer estimation APIs.
_{🤖 Reviewed with Codex}

[kriszyp]

This is still not using the estimation functionality in rocksdb 2.8.
_{🤖 Reviewed with Codex}

[kriszyp]

Can you rebase/stack this onto #2163, which owns the implementation of using estimation for query planning, deleting rangeBoundsForEstimate / estimateRangeCount / the RocksIndexStore override here, along with the dep bump. #2147 then keeps what's genuinely its own — Prefer: count=, the guardrails, the Content-Range headers, the page materialization — and the estimateCondition() call inherits the corrected estimates for free.

rocksdb-js 2.8.0 also ships CountEstimator (db.createCountEstimator()), which progressively refines a range estimate from entries already iterated. The count=estimated path is a natural fit — it walks offset + limit entries of the range and then discards that knowledge for a pure statistical estimate. Feeding it advance(lastKey, n) per page makes the traversed portion exact and calibrates the remainder, and finish() on natural exhaustion returns confidence: 1 — an exact total for free whenever the match set fits inside the guardrail, which is most paginated queries. That would also subsume a good chunk of the count=exact machinery. We could potentially do this as a follow-up, but I think it makes more sense here.

_{🤖 Reviewed with Codex}

[cb1kenobi]

@kriszyp — done, and thanks for the detailed steer through this. Summary of where things landed:

Rebased onto #2163. This branch is merged onto main, which now carries #2163 (the query-planner estimation work). I deleted rangeBoundsForEstimate / estimateRangeCount / the RocksIndexStore.estimateCount override and the rocksdb-js dep bump from this PR, so #2147 keeps only what's genuinely its own — Prefer: count=, the guardrails, the Content-Range headers, and page materialization — and estimateCondition() inherits #2163's storage-level range estimates for count=estimated. Net diff vs main in search.ts is now a single line (re-exporting estimatedEntryCount, which the count path imports).

HNSW/vector count=exact blocker fixed (9e44500). An approximate result set — a vector/HNSW-driven sort or a vectorFilter — now reports its total as unavailable (items x-y/*) rather than advertising a page-size-dependent scanned as exact. Regression test runs the same cosine-sorted query at two page sizes and asserts the total is unavailable at both.

The remaining inline threads (GET/HEAD-only preference, Vary: Prefer, CORS token comparison, limit(Infinity)/offset guards, exact-count opt-in) were addressed in earlier commits — I replied inline on each with the specifics and resolved them.

Full resources unit suite is green (1883). Ready for another look when you have a moment!

_{🤖 Drafted with Claude Code (Opus 4.8)}

[cb1kenobi]

Cross-model review (Gemini 3.1 Pro + Cursor-Grok)

Ran the pre-push cross-model review on d7d4356b6. The Codex graded leg timed out on a local model-manager error, which starved the domain adjudicator, so findings below are author-triaged, not machine-adjudicated. Independent outside-family coverage still came from Gemini and Cursor-Grok. Verified each against the code before acting.

Fixed (pushed in d7d4356b6)

• Event-loop starvation on a large count=exact scan (Gemini major / Grok significant). The drain iterated the store's async iterator with no macrotask yield; on a synchronously-settling iterator (the common indexed-scan case) a big scan ran as one microtask burst, blocking the loop for up to the whole MAX_EXACT_COUNT_MS. It now yields to the macrotask queue every COUNT_YIELD_INTERVAL rows — covering the page-window scan too, not just the tail.
• Approximate exact-count wasted the tail (Grok suggestion). For a vector/HNSW result set, count=exact now stops at the page window instead of draining a tail whose total is reported unavailable anyway.
• Prefer-parse hardening (Gemini minors). Optional-chain httpOptions (a programmatic mount may pass none) and String()-coerce the preference value before lower-casing (a bare Prefer: count with no = can yield a non-string).

Investigated — not a bug

• "async allowRead breaks the count path → 500" (Grok blocker). Did not reproduce: driving get() (what REST calls) with an async allowRead and Prefer: count= returns a proper page array — recordCount intact, iterates cleanly. The Table-level async-authorization branch isn't reached on the count path (authorization resolves at the Resource layer first). No fix shipped for a non-issue.

Surfaced for a maintainer call (not changed)

These are real but touch product/security defaults, so I'd rather you decide than change them unilaterally:

• CORS request admission (Grok significant): the default Access-Control-Allow-Headers (Accept, Content-Type, Authorization) omits Prefer, so a cross-origin browser can't send Prefer — preflight blocks the request before the new expose-headers path matters. The feature's cross-origin story is only half-wired without it. Options: add Prefer to the default allow-list (small, safe — Prefer is a standard header), or document that operators must. Happy to implement either.
• Conditional GET + count (Grok significant): a matching If-None-Match returns 304 and clears the body after the count scan already ran, so Content-Range/Preference-Applied are dropped and the scan was wasted. Edge (needs a prior ETag), but worth a decision: skip the scan on a 304, or emit the count headers on it.
• Quoted preference token (Grok suggestion): Prefer: count="exact" (RFC-quoted) is silently ignored. Cheap to accept if we want it.
• Minor (Gemini): setCountHeaders allocates a Set/array per count response (opt-in path, low impact); and the new comments run heavier than Harper's usual density — can trim on request.

Full resources unit suite: 1876 passing; the 8 failures (transaction-log / snapshot / audit / table-reload) reproduce identically on a clean tree — pre-existing and unrelated to this change.

_{🤖 Drafted with Claude Code (Opus 4.8)}

[kriszyp]

Awesome, great work!
_{🤖 Reviewed with Codex}

[kriszyp]

approximateResultSet only examines target.vectorFilter and the sort chain, but HNSW is also directly queryable through lt/le conditions. For example, conditions: [{ attribute: 'vector', comparator: 'lt', value: threshold, target }] has neither marker, so this remains false even though executeConditions() passes the page-derived minResults into the same bounded HNSW candidate search.

When that candidate iterator exhausts, exact remains true and line 3787 still publishes scanned as an authoritative total. Please include HNSW-backed planned conditions in the approximate-result classification and add a threshold-condition regression alongside the vector-sort test.

— KrAIs (GPT-5)