From 81219d412214273048e628bbe347b4d206459395 Mon Sep 17 00:00:00 2001 From: Chris Barber Date: Tue, 11 Aug 2026 02:30:57 -0500 Subject: [PATCH 1/6] docs(rest): pagination total count (Prefer: count=) and exactCount option MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Documents the REST pagination total-count feature shipping in Harper v5.3.0: - reference/rest/querying.md: new "Pagination and Total Count" section covering Prefer: count=exact|estimated, the Content-Range / Range-Unit / Preference-Applied response headers, the unavailable-total (.../*) case, HEAD pre-flight, CORS exposure, and disabling exact counts per mount. - reference/rest/overview.md: adds the `exactCount` rest-mount option. - reference/rest/headers.md: adds the Prefer request header and notes the count response headers. Version badges assume v5.3.0 (next minor after 5.2) — adjust if the feature lands in a different release. Pairs with the harper core branch feat/rest-pagination-total-count. Co-Authored-By: Claude Opus 4.8 (1M context) --- reference/rest/headers.md | 15 ++++++++++++ reference/rest/overview.md | 3 +++ reference/rest/querying.md | 49 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 67 insertions(+) diff --git a/reference/rest/headers.md b/reference/rest/headers.md index fea04b908..95239e892 100644 --- a/reference/rest/headers.md +++ b/reference/rest/headers.md @@ -20,6 +20,8 @@ These headers are included in all Harper REST API responses: | `etag` | `"abc123"` | Encoded version/last-modification time of the returned record. Used for conditional requests. | | `location` | `/MyTable/new-id` | Returned on `POST` responses. Contains the path to the newly created record. | +Collection responses to a [count request](./querying.md#pagination-and-total-count) additionally include `Content-Range`, `Range-Unit`, and `Preference-Applied` (). + ## Cache-Control @@ -100,6 +102,19 @@ Accept-Encoding: gzip, br Compression is particularly effective for JSON responses. For binary formats like CBOR, compression provides diminishing returns compared to the already-compact encoding. +### Prefer + + + +Opt in to a total match count on a collection `GET`/`HEAD`, returned via the `Content-Range` response header for pagination: + +```http +GET /Product/?category=software&limit(0,25) +Prefer: count=exact +``` + +Accepts `count=exact` (precise, scans the full matched set) or `count=estimated` (fast, approximate). See [Pagination and Total Count](./querying.md#pagination-and-total-count) for the full request/response contract. + ### Authorization Credentials for authenticating requests. See [Security Overview](../security/overview.md) for details on supported authentication mechanisms (Basic, JWT, mTLS). diff --git a/reference/rest/overview.md b/reference/rest/overview.md index 252d02265..0c9b3c16e 100644 --- a/reference/rest/overview.md +++ b/reference/rest/overview.md @@ -33,8 +33,11 @@ rest: true rest: lastModified: true # enables Last-Modified response header support webSocket: false # disables automatic WebSocket support (enabled by default) + exactCount: false # serve Prefer: count=exact requests as estimates instead of scanning ``` + `exactCount` (default `false`) controls whether the [pagination total-count](./querying.md#pagination-and-total-count) feature honors `Prefer: count=exact`. Because an exact count scans the full matched set, it is off by default and such requests are served as cheaper estimates; set it to `true` for the REST interface to enable exact counts. + ## Tables and Their Automatic Endpoints This section describes the **default table Resource** — the endpoints Harper registers automatically for a table, with no handler code of your own. Harper serves that default Resource only when **both** of the following are true: diff --git a/reference/rest/querying.md b/reference/rest/querying.md index a26082c48..20c40cf79 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -173,6 +173,55 @@ GET /Product/?rating=gt=3&sort(+name) GET /Product/?sort(+rating,-price) ``` +## Pagination and Total Count + + + +Use `limit(start,end)` to page through a collection, and opt in to a total match count with the `Prefer: count=` request header so a client can render pagination (for example "1-25 of 1,234") without a second request. + +Counting is opt-in: without the header, no count is computed and no count headers are returned. + +### Requesting a count + +Send a `Prefer` header on a `GET` (or `HEAD`) request to a collection: + +| Value | Meaning | +| ----------------- | --------------------------------------------------------------------------------------------------------------- | +| `count=exact` | The exact number of matching records. Scans the full matched set, so it is more expensive than the page itself. | +| `count=estimated` | A fast planner/table estimate. Cheap, approximate. | + +```http +GET /Product/?category=software&limit(0,25) +Prefer: count=exact +``` + +### Count response headers + +The count is returned in [RFC 7233](https://datatracker.ietf.org/doc/html/rfc7233)-style response headers alongside the page body: + +| Header | Example | Description | +| -------------------- | ----------------- | --------------------------------------------------------------------------------------------- | +| `Content-Range` | `items 0-24/1234` | The 0-based, inclusive range of records returned (`start-end`) out of the total matching set. | +| `Range-Unit` | `items` | The unit used by `Content-Range`. | +| `Preference-Applied` | `count=exact` | The count mode the server applied (`exact` or `estimated`). | + +```http +HTTP/1.1 200 OK +Content-Range: items 0-24/1234 +Range-Unit: items +Preference-Applied: count=exact +``` + +The response status is always `200` — `Content-Range` is informational (Harper does not use `206 Partial Content`). A `HEAD` request with `Prefer: count=` returns the count headers with no body, a cheap way to ask "how many match?" without transferring the page. When CORS is enabled, these three headers are added to `Access-Control-Expose-Headers` so browser clients can read them cross-origin. + +### Unavailable totals + +The total is reported as `*` (for example `Content-Range: items 0-24/*`) when it cannot be produced — an exact scan that reaches its internal work limit, or a query with no cardinality estimate (for example a `!=` or `contains` condition). `Preference-Applied` still echoes the requested mode, so an unavailable total (`.../*`) is distinct from a request that asked for no count. + +### Disabling exact counts + +Because an exact count scans the full matched set, a deployment can disable it per REST mount via the [`exactCount` option](./overview.md#configuration). With `exactCount: false`, a `count=exact` request is served as an estimate instead (the response reports `Preference-Applied: count=estimated`). Estimated counts are always available; the default is `true`. + ## Relationships and Joins From 210ab87514b464be991400d507c54ab4d7ce04f7 Mon Sep 17 00:00:00 2001 From: Chris Barber Date: Tue, 11 Aug 2026 18:34:18 -0500 Subject: [PATCH 2/6] docs(rest): note that pagination count requires a limit() A count request without a limit() is served normally with no count headers (the core feature falls through to streaming rather than counting the whole collection). Co-Authored-By: Claude Opus 4.8 (1M context) --- reference/rest/querying.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/rest/querying.md b/reference/rest/querying.md index 20c40cf79..4007dd634 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -179,7 +179,7 @@ GET /Product/?sort(+rating,-price) Use `limit(start,end)` to page through a collection, and opt in to a total match count with the `Prefer: count=` request header so a client can render pagination (for example "1-25 of 1,234") without a second request. -Counting is opt-in: without the header, no count is computed and no count headers are returned. +Counting is opt-in: without the header, no count is computed and no count headers are returned. It also requires a `limit()` — a count request on an unbounded collection (no `limit()`) is served normally, with no count headers, since counting the whole collection would defeat the point of paging. ### Requesting a count From 34f2f01763439ed4e93d97abee2c0c07cbe8bc7f Mon Sep 17 00:00:00 2001 From: Chris Barber Date: Tue, 11 Aug 2026 21:58:54 -0500 Subject: [PATCH 3/6] docs(rest): clarify exactCount scope wording; small querying nits MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Addresses review feedback on #623: - Drop the confusing/inaccurate "per REST mount" / "on a given mount" scope wording for exactCount. It is not a global setting — it is read only from a component's `rest:` config (server/REST.ts), so describe it as configured "in an application's REST configuration" rather than a mount or a global option. - Refer to the header as the `Prefer` request header (`Prefer: count=exact`). - Use the `=ct=` operator spelling for the contains example, matching the operators table. Co-Authored-By: Claude Opus 4.8 (1M context) --- reference/rest/overview.md | 2 +- reference/rest/querying.md | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/reference/rest/overview.md b/reference/rest/overview.md index 0c9b3c16e..abe0f3391 100644 --- a/reference/rest/overview.md +++ b/reference/rest/overview.md @@ -36,7 +36,7 @@ rest: exactCount: false # serve Prefer: count=exact requests as estimates instead of scanning ``` - `exactCount` (default `false`) controls whether the [pagination total-count](./querying.md#pagination-and-total-count) feature honors `Prefer: count=exact`. Because an exact count scans the full matched set, it is off by default and such requests are served as cheaper estimates; set it to `true` for the REST interface to enable exact counts. + `exactCount` (default `false`) controls whether the [pagination total-count](./querying.md#pagination-and-total-count) feature honors `Prefer: count=exact`. Because an exact count scans the full matched set, it is off by default and such requests are served as cheaper estimates; set it to `true` for this application's REST interface to enable exact counts. ## Tables and Their Automatic Endpoints diff --git a/reference/rest/querying.md b/reference/rest/querying.md index 4007dd634..0c66751ae 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -177,7 +177,7 @@ GET /Product/?sort(+rating,-price) -Use `limit(start,end)` to page through a collection, and opt in to a total match count with the `Prefer: count=` request header so a client can render pagination (for example "1-25 of 1,234") without a second request. +Use `limit(start,end)` to page through a collection, and opt in to a total match count with the `Prefer` request header (`Prefer: count=exact`) so a client can render pagination (for example "1-25 of 1,234") without a second request. Counting is opt-in: without the header, no count is computed and no count headers are returned. It also requires a `limit()` — a count request on an unbounded collection (no `limit()`) is served normally, with no count headers, since counting the whole collection would defeat the point of paging. @@ -216,11 +216,11 @@ The response status is always `200` — `Content-Range` is informational (Harper ### Unavailable totals -The total is reported as `*` (for example `Content-Range: items 0-24/*`) when it cannot be produced — an exact scan that reaches its internal work limit, or a query with no cardinality estimate (for example a `!=` or `contains` condition). `Preference-Applied` still echoes the requested mode, so an unavailable total (`.../*`) is distinct from a request that asked for no count. +The total is reported as `*` (for example `Content-Range: items 0-24/*`) when it cannot be produced — an exact scan that reaches its internal work limit, or a query with no cardinality estimate (for example a `!=` or `=ct=` (contains) condition). `Preference-Applied` still echoes the requested mode, so an unavailable total (`.../*`) is distinct from a request that asked for no count. ### Disabling exact counts -Because an exact count scans the full matched set, a deployment can disable it per REST mount via the [`exactCount` option](./overview.md#configuration). With `exactCount: false`, a `count=exact` request is served as an estimate instead (the response reports `Preference-Applied: count=estimated`). Estimated counts are always available; the default is `true`. +Because an exact count scans the full matched set, you can disable it in an application's REST configuration via the [`exactCount` option](./overview.md#configuration). With `exactCount: false`, a `count=exact` request is served as an estimate instead (the response reports `Preference-Applied: count=estimated`). Estimated counts are always available; the default is `true`. From 7e11104b9b3db217715cd81e5e50bd773028e207 Mon Sep 17 00:00:00 2001 From: Chris Barber Date: Thu, 13 Aug 2026 11:43:33 -0500 Subject: [PATCH 4/6] docs(rest): exact counting is opt-in (default off); GET/HEAD + bounded limit Follows the harper #2147 review outcome: exact counting is now off by default and enabled per mount with `rest: { exactCount: true }`; count=exact is otherwise served as an estimate. Also note that counting applies to GET/HEAD only and requires a limit() within a supported page size. Co-Authored-By: Claude Opus 4.8 (1M context) --- reference/rest/overview.md | 4 ++-- reference/rest/querying.md | 21 ++++++++++++++------- 2 files changed, 16 insertions(+), 9 deletions(-) diff --git a/reference/rest/overview.md b/reference/rest/overview.md index abe0f3391..0f60b27c6 100644 --- a/reference/rest/overview.md +++ b/reference/rest/overview.md @@ -33,10 +33,10 @@ rest: true rest: lastModified: true # enables Last-Modified response header support webSocket: false # disables automatic WebSocket support (enabled by default) - exactCount: false # serve Prefer: count=exact requests as estimates instead of scanning + exactCount: true # opt in to Prefer: count=exact scans (off by default; otherwise served as estimates) ``` - `exactCount` (default `false`) controls whether the [pagination total-count](./querying.md#pagination-and-total-count) feature honors `Prefer: count=exact`. Because an exact count scans the full matched set, it is off by default and such requests are served as cheaper estimates; set it to `true` for this application's REST interface to enable exact counts. + `exactCount` (default `false`) controls whether the [pagination total-count](./querying.md#pagination-and-total-count) feature honors `Prefer: count=exact`. Because an exact count scans the full matched set, it is off by default; set this to `true` to enable exact counts for this application's REST interface. A `count=exact` request is otherwise served as a cheaper estimate. ## Tables and Their Automatic Endpoints diff --git a/reference/rest/querying.md b/reference/rest/querying.md index 0c66751ae..7bf942c96 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -179,16 +179,16 @@ GET /Product/?sort(+rating,-price) Use `limit(start,end)` to page through a collection, and opt in to a total match count with the `Prefer` request header (`Prefer: count=exact`) so a client can render pagination (for example "1-25 of 1,234") without a second request. -Counting is opt-in: without the header, no count is computed and no count headers are returned. It also requires a `limit()` — a count request on an unbounded collection (no `limit()`) is served normally, with no count headers, since counting the whole collection would defeat the point of paging. +Counting is opt-in: without the header, no count is computed and no count headers are returned. It applies only to `GET`/`HEAD` requests and requires a `limit()` within a supported page size — a request with no `limit()`, an oversized one, or a non-numeric one is served normally with no count headers, since counting an unbounded page would defeat the point of paging. ### Requesting a count Send a `Prefer` header on a `GET` (or `HEAD`) request to a collection: -| Value | Meaning | -| ----------------- | --------------------------------------------------------------------------------------------------------------- | -| `count=exact` | The exact number of matching records. Scans the full matched set, so it is more expensive than the page itself. | -| `count=estimated` | A fast planner/table estimate. Cheap, approximate. | +| Value | Meaning | +| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | +| `count=exact` | The exact number of matching records. Scans the full matched set, so it is opt-in per mount (see below) and served as an estimate unless enabled. | +| `count=estimated` | A fast planner/table estimate. Cheap, approximate. | ```http GET /Product/?category=software&limit(0,25) @@ -218,9 +218,16 @@ The response status is always `200` — `Content-Range` is informational (Harper The total is reported as `*` (for example `Content-Range: items 0-24/*`) when it cannot be produced — an exact scan that reaches its internal work limit, or a query with no cardinality estimate (for example a `!=` or `=ct=` (contains) condition). `Preference-Applied` still echoes the requested mode, so an unavailable total (`.../*`) is distinct from a request that asked for no count. -### Disabling exact counts +### Enabling exact counts -Because an exact count scans the full matched set, you can disable it in an application's REST configuration via the [`exactCount` option](./overview.md#configuration). With `exactCount: false`, a `count=exact` request is served as an estimate instead (the response reports `Preference-Applied: count=estimated`). Estimated counts are always available; the default is `true`. +Because an exact count scans the full matched set, it is **off by default**. Enable it in an application's REST configuration via the [`exactCount` option](./overview.md#configuration): + +```yaml +rest: + exactCount: true +``` + +Without it, a `count=exact` request is served as an estimate (the response reports `Preference-Applied: count=estimated`). Estimated counts are always available. From a145b69811c145d20da40c83b0ef45ada16c54f6 Mon Sep 17 00:00:00 2001 From: Chris Barber Date: Wed, 19 Aug 2026 10:28:32 -0500 Subject: [PATCH 5/6] Update reference/rest/headers.md Co-authored-by: Dawson Toth --- reference/rest/headers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/rest/headers.md b/reference/rest/headers.md index 95239e892..f0a98c215 100644 --- a/reference/rest/headers.md +++ b/reference/rest/headers.md @@ -110,7 +110,7 @@ Opt in to a total match count on a collection `GET`/`HEAD`, returned via the `Co ```http GET /Product/?category=software&limit(0,25) -Prefer: count=exact +Prefer: count=estimated ``` Accepts `count=exact` (precise, scans the full matched set) or `count=estimated` (fast, approximate). See [Pagination and Total Count](./querying.md#pagination-and-total-count) for the full request/response contract. From 28bd76e33584032a25ddaec28a45d62281e77648 Mon Sep 17 00:00:00 2001 From: Chris Barber Date: Wed, 2 Sep 2026 17:26:22 -0500 Subject: [PATCH 6/6] docs(rest): address review feedback on pagination count MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Document the count page-size bounds (limit ≤ 10,000, window ≤ 1,000,000) and name the exact-scan guardrail (1M-row / ~1s budget) that yields an unavailable total. - Frame HEAD + count=exact as bandwidth-saving, not a low-cost counting shortcut (still scans the matched set). - Clarify estimated counts don't require exactCount, and their total can still be unavailable (*). - Scope exactCount to the REST interface rather than a per-mount setting. Co-Authored-By: Claude Opus 4.8 (1M context) --- reference/rest/querying.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/reference/rest/querying.md b/reference/rest/querying.md index 7bf942c96..b8d7d0e66 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -179,16 +179,16 @@ GET /Product/?sort(+rating,-price) Use `limit(start,end)` to page through a collection, and opt in to a total match count with the `Prefer` request header (`Prefer: count=exact`) so a client can render pagination (for example "1-25 of 1,234") without a second request. -Counting is opt-in: without the header, no count is computed and no count headers are returned. It applies only to `GET`/`HEAD` requests and requires a `limit()` within a supported page size — a request with no `limit()`, an oversized one, or a non-numeric one is served normally with no count headers, since counting an unbounded page would defeat the point of paging. +Counting is opt-in: without the header, no count is computed and no count headers are returned. It applies only to `GET`/`HEAD` requests and requires a bounded page — the `limit()` must be a non-negative integer no larger than **10,000**, and the requested window (offset + limit) no larger than **1,000,000**. A request with no `limit()`, one outside those bounds, or a non-numeric one is served normally with no count headers, since counting an unbounded page would defeat the point of paging. ### Requesting a count Send a `Prefer` header on a `GET` (or `HEAD`) request to a collection: -| Value | Meaning | -| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | -| `count=exact` | The exact number of matching records. Scans the full matched set, so it is opt-in per mount (see below) and served as an estimate unless enabled. | -| `count=estimated` | A fast planner/table estimate. Cheap, approximate. | +| Value | Meaning | +| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `count=exact` | The exact number of matching records. Scans the full matched set, so it is off by default (see [below](#enabling-exact-counts)) and served as an estimate unless enabled for the REST interface. | +| `count=estimated` | A fast planner/table estimate. Cheap, approximate. | ```http GET /Product/?category=software&limit(0,25) @@ -212,11 +212,11 @@ Range-Unit: items Preference-Applied: count=exact ``` -The response status is always `200` — `Content-Range` is informational (Harper does not use `206 Partial Content`). A `HEAD` request with `Prefer: count=` returns the count headers with no body, a cheap way to ask "how many match?" without transferring the page. When CORS is enabled, these three headers are added to `Access-Control-Expose-Headers` so browser clients can read them cross-origin. +The response status is always `200` — `Content-Range` is informational (Harper does not use `206 Partial Content`). A `HEAD` request with a `count` preference returns the count headers with no body — it saves transferring the page, but `count=exact` still scans the matched set (subject to the same guardrails), so it is a bandwidth-saving pre-flight, not a low-cost counting shortcut. When CORS is enabled, these three headers are added to `Access-Control-Expose-Headers` so browser clients can read them cross-origin. ### Unavailable totals -The total is reported as `*` (for example `Content-Range: items 0-24/*`) when it cannot be produced — an exact scan that reaches its internal work limit, or a query with no cardinality estimate (for example a `!=` or `=ct=` (contains) condition). `Preference-Applied` still echoes the requested mode, so an unavailable total (`.../*`) is distinct from a request that asked for no count. +The total is reported as `*` (for example `Content-Range: items 0-24/*`) when it cannot be produced — an exact scan that hits its guardrail (counting the tail past the requested page is bounded by a 1,000,000-row cap and a ~1-second budget, and abandons the total rather than truncating the page), or a query with no cardinality estimate (for example a `!=` or `=ct=` (contains) condition). `Preference-Applied` still echoes the requested mode, so an unavailable total (`.../*`) is distinct from a request that asked for no count. ### Enabling exact counts @@ -227,7 +227,7 @@ rest: exactCount: true ``` -Without it, a `count=exact` request is served as an estimate (the response reports `Preference-Applied: count=estimated`). Estimated counts are always available. +Without it, a `count=exact` request is served as an estimate (the response reports `Preference-Applied: count=estimated`). Estimated counts do not require `exactCount`; when no estimate is available, the total is reported as `*`.