feat(fumadb): add aggregate and keyset query support

[aryasaatvik]

Summary

Add policy-scoped JSON-document aggregation and keyset pagination to FumaDB, with memory and Drizzle implementations, and expose both through indexed plugin-storage collections.

API

const byStatus = yield* runs.aggregate.groupCount({ field: "status" });
const latency = yield* runs.aggregate.stats({
  field: "durationMs",
  percentiles: [0.5, 0.95],
});
const page = yield* runs.queryKeyset({
  where: { status: "completed" },
  orderBy: [{ field: "startedAt", direction: "desc", valueType: "number" }],
  limit: 50,
});

plugin collection aggregate/queryKeyset
  -> validate declared indexed fields
  -> translate filters and apply table read policies
  -> FumaDB JSON aggregation or keyset page
  -> memory evaluation or SQLite/Postgres pushdown

• Memory and SQL adapters share filter and comparison semantics.
• Drizzle handles quoted paths, escaped LIKE, SQL nulls, percentiles, and nullable cursors.
• Plugin-storage keys provide deterministic page tiebreakers.
• Unsupported adapter capabilities fail explicitly.

Validation

• Full FumaDB, SDK, and OpenAPI package suites passed.
• Memory and SQLite cover aggregates, policies, filter parity, percentiles, and nullable keysets.
• Affected package typechecks, format, and lint passed.
• Postgres SQL generation is not yet exercised by a local Postgres harness.

Execution history delivery map

Prerequisites:

• feat(execution): add execution observer foundation #1097 — Execution observer foundation
• feat(fumadb): add aggregate and keyset query support #1119 — Aggregate and keyset queries 👈 current

Follow-up PR-sized diffs:

• Core persistence and read API
  • Runs console UI
    • Cloudflare service-token actors

Execution history is the proven consumer for this query surface, not part of this PR. View the complete fork comparison.

[greptile-apps]

Greptile Summary

This PR adds SQL-pushed JSON-document aggregate primitives and keyset pagination to FumaDB, exposing them through the plugin storage SDK facade. All previous review concerns (empty or filter semantics, explicit operator mapping, SQLite percentile documentation) have been addressed in this revision.

• FumaDB core: New jsonCount, jsonGroupCount, jsonTimeBuckets, jsonStats, and jsonPage methods on AbstractQuery, backed by shared in-memory evaluation helpers and Drizzle SQLite/Postgres pushdown with null-aware keyset cursor predicates and dialect-specific JSON path quoting/LIKE escaping.
• Plugin storage facade: New aggregate.{count,groupCount,timeBuckets,stats} and queryKeyset(...) methods on collection facades, translating indexed-field where conditions into JsonFilter predicates while table-read policies enforce owner/tenant scoping before every adapter call.
• Test coverage: Memory and SQLite harnesses exercising count, group count, time buckets, stats, keyset pagination with null sort values, LIKE-wildcard parity, path-quoting edge cases, and policy enforcement for the new operations.

Confidence Score: 5/5

Safe to merge — all five new query operations apply table read policies before reaching the adapter, and the empty-or / explicit-operator issues from the previous review are resolved.

The implementation is correct and well-tested across memory, SQLite, and policy harnesses. The null-aware keyset cursor logic is exercised end-to-end (including the nullable-sort-column truncation regression). JSON path quoting, LIKE escaping, and empty composite filter semantics all have explicit parity tests between the memory and Drizzle adapters.

No files require special attention. The most complex file (drizzle/query.ts) is fully covered by the aggregate.test.ts harness for SQLite, and the Postgres-specific paths (percentile_cont, nulls last/first) are intentionally deferred to a future Postgres test harness as noted in the PR description.

Important Files Changed

Filename	Overview
packages/core/fumadb/src/adapters/drizzle/query.ts	Adds jsonCount/jsonGroupCount/jsonTimeBuckets/jsonStats/jsonPage adapter hooks with dialect-aware JSON path quoting, LIKE wildcard escaping with correct ESCAPE clause, null-aware keyset cursor predicates, and Postgres percentile_cont pushdown. Empty or filter correctly returns 1 = 0.
packages/core/fumadb/src/query/orm/index.ts	Adds compileScopedWhere helper that applies read policies before every new JSON operation, requireJsonOp that throws clearly for unsupported adapters. The scopedWhere === false short-circuit correctly returns zero/empty results when a policy statically denies access.
packages/core/sdk/src/executor.ts	Adds CoreDb plumbing for the five JSON operations, pluginStorageWhereToJsonFilter translation, pluginStorageInvalidLimitError guard, and the queryKeyset/aggregate facade implementations. nextCursor is correctly set only when entries.length >= limit. pluginStorageJsonCompareOperators is now an explicit mapping.
packages/core/fumadb/src/query/aggregate-eval.ts	New file with pure in-memory evaluation helpers: extractJsonPath, coerceJsonValue, matchesJsonFilter, bucketFloor, computePercentiles (Postgres percentile_cont linear interpolation), and compareNullableAscending.
packages/core/fumadb/src/adapters/memory/index.ts	Adds all five JSON operations using the shared aggregate-eval helpers. The jsonPage implementation correctly applies null-aware compareNullableAscending sort and cursor filter, consistent with the Drizzle adapter's SQL null ordering semantics.
packages/core/fumadb/src/query/aggregate.ts	New public type definitions for all JSON aggregate and keyset pagination primitives, split into adapter-facing options and public options. Clean separation of concerns.
packages/core/fumadb/src/query/aggregate.test.ts	New test file running the full aggregate + keyset suite against memory and SQLite harnesses, covering null SQL three-valued logic, empty composite filters, nullable sort column keyset pagination, LIKE wildcard parity, and path-quoting edge cases.
packages/core/sdk/src/plugin-storage-aggregate.test.ts	New integration test running the full plugin storage aggregate + keyset suite against the SQLite pushdown path, including multi-page keyset cursor pagination with a JSON filter.
packages/core/fumadb/src/query/table-policy.test.ts	Extended with a test confirming read policies are applied before all JSON aggregate and keyset operations, and that a denied context correctly returns zero count and empty page results.
packages/core/sdk/src/plugin-storage.ts	Adds public facade types: PluginStorageAggregateFilter, PluginStorageQueryKeysetInput, PluginStorageKeysetPage, and PluginStorageAggregateFacade. queryKeyset and aggregate added to PluginStorageCollectionFacade.
packages/core/sdk/src/fuma-runtime.ts	Adds the five new JSON operation forwards to makeSafeFumaQuery, consistent with the existing operation forwarding pattern.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Plugin as Plugin Code
    participant Facade as PluginStorageFacade
    participant CoreDb as CoreDb
    participant ORM as toORM (orm/index.ts)
    participant Adapter as Drizzle/Memory Adapter

    Plugin->>Facade: "storage.runs.aggregate.groupCount({ field, where })"
    Facade->>Facade: validate indexed fields
    Facade->>Facade: pluginStorageWhereToJsonFilter(where)
    Facade->>CoreDb: "jsonGroupCount(plugin_storage, { column, where, filter })"
    CoreDb->>ORM: jsonGroupCount(name, options)
    ORM->>ORM: requireJsonOp + compileScopedWhere
    ORM->>ORM: applyReadPolicies
    ORM->>Adapter: "jsonGroupCount(table, { column, where, filter, path })"
    Adapter-->>ORM: JsonGroupCountRow[]
    ORM-->>Facade: JsonGroupCountRow[]
    Facade-->>Plugin: PluginStorageGroupCount[]

    Plugin->>Facade: "storage.runs.queryKeyset({ orderBy, cursor, limit })"
    Facade->>Facade: validate indexed fields + limit
    Facade->>CoreDb: "jsonPage(plugin_storage, { orderBy, keyColumn, cursor, limit })"
    CoreDb->>ORM: jsonPage(name, options)
    ORM->>ORM: requireJsonOp + compileScopedWhere
    ORM->>ORM: applyReadPolicies
    ORM->>Adapter: "jsonPage(table, { orderBy, keyColumn, keyDirection, cursor, limit })"
    Note over Adapter: null-aware cursor predicate OR-terms
    Adapter-->>ORM: Row[]
    ORM-->>Facade: Row[]
    Facade->>Facade: build nextCursor from last entry
    Facade-->>Plugin: "{ entries, nextCursor }"

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Plugin as Plugin Code
    participant Facade as PluginStorageFacade
    participant CoreDb as CoreDb
    participant ORM as toORM (orm/index.ts)
    participant Adapter as Drizzle/Memory Adapter

    Plugin->>Facade: "storage.runs.aggregate.groupCount({ field, where })"
    Facade->>Facade: validate indexed fields
    Facade->>Facade: pluginStorageWhereToJsonFilter(where)
    Facade->>CoreDb: "jsonGroupCount(plugin_storage, { column, where, filter })"
    CoreDb->>ORM: jsonGroupCount(name, options)
    ORM->>ORM: requireJsonOp + compileScopedWhere
    ORM->>ORM: applyReadPolicies
    ORM->>Adapter: "jsonGroupCount(table, { column, where, filter, path })"
    Adapter-->>ORM: JsonGroupCountRow[]
    ORM-->>Facade: JsonGroupCountRow[]
    Facade-->>Plugin: PluginStorageGroupCount[]

    Plugin->>Facade: "storage.runs.queryKeyset({ orderBy, cursor, limit })"
    Facade->>Facade: validate indexed fields + limit
    Facade->>CoreDb: "jsonPage(plugin_storage, { orderBy, keyColumn, cursor, limit })"
    CoreDb->>ORM: jsonPage(name, options)
    ORM->>ORM: requireJsonOp + compileScopedWhere
    ORM->>ORM: applyReadPolicies
    ORM->>Adapter: "jsonPage(table, { orderBy, keyColumn, keyDirection, cursor, limit })"
    Note over Adapter: null-aware cursor predicate OR-terms
    Adapter-->>ORM: Row[]
    ORM-->>Facade: Row[]
    Facade->>Facade: build nextCursor from last entry
    Facade-->>Plugin: "{ entries, nextCursor }"

Loading

_{Reviews (3): Last reviewed commit: "fix(fumadb): preserve empty or filter se..." | Re-trigger Greptile}

[aryasaatvik]

Rebased onto current main and refreshed at c9e1fb3. Branch-focused FumaDB, SDK, and OpenAPI tests; full package suites; format; lint; typecheck; unit tests; and 35 of 36 current GitHub checks pass. The sole red check is unrelated selfhost E2E infrastructure: admin-users-pager.test.ts uses a global getByRole button query for Next, which matched both the pager button and a generated user-row button named Next, causing a Playwright strict-mode failure. The affected storage tests and the previously failing OAuth scenario pass locally.