Skip to content

perf(storage): add bulk plugin-storage upserts - #1098

Merged
RhysSullivan merged 14 commits into
UsefulSoftwareCo:mainfrom
aryasaatvik:contrib/plugin-storage-bulk-writes
Aug 28, 2026
Merged

perf(storage): add bulk plugin-storage upserts#1098
RhysSullivan merged 14 commits into
UsefulSoftwareCo:mainfrom
aryasaatvik:contrib/plugin-storage-bulk-writes

Conversation

@aryasaatvik

@aryasaatvik aryasaatvik commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

Summary

Add policy-safe bulk upserts to FumaDB and route plugin-storage batch writes through conflict upserts instead of delete-then-create loops.

API

yield* collection.putMany({ owner: "org", entries });
yield* collection.removeMany({ owner: "org", keys });
collection.putMany
  -> deduplicate keys and bind the owner partition
  -> enter the FumaDB transaction boundary
  -> enforce table policies
  -> memory upsert or parameter-bounded Drizzle conflict upsert
  • AbstractQuery and the ORM adapter gain single-row and bulk upserts.
  • Organization/user ownership and read precedence remain unchanged.
  • Interactive adapters use transactions; D1 uses its native transactional batch() API.
  • Unsupported adapters fail explicitly.

Proven consumers

The fork uses bulk writes for semantic-search indexing and OpenAPI operation replacement. This is an independent storage optimization, not a prerequisite for execution history.

Validation

  • FumaDB suite: 4 files, 35 tests.
  • SDK plugin-storage suite: 6 tests.
  • D1 rollback coverage proves later batch failures commit no earlier rows.
  • SDK rollback coverage proves partial adapter writes are rolled back by PluginStorage.putMany.

JSON aggregation, keyset pagination, and semantic-search behavior remain separate changes.

@aryasaatvik
aryasaatvik marked this pull request as ready for review June 24, 2026 06:20
@aryasaatvik
aryasaatvik force-pushed the contrib/plugin-storage-bulk-writes branch from f2a006d to 5851d0a Compare June 24, 2026 06:20
@greptile-apps

greptile-apps Bot commented Jun 24, 2026

Copy link
Copy Markdown

Greptile Summary

This PR adds a upsertMany primitive to FumaDB's query/adapter stack and routes plugin-storage bulk writes through conflict-aware INSERT … ON CONFLICT DO UPDATE statements, replacing the previous delete-then-insert pattern. Policy enforcement (create + update) is applied per-row at the ORM layer before any adapter call, and permitted rows are grouped by their policy condition key for efficient native batching.

  • FumaDB layer: AbstractQuery.upsertMany is fully wired through the ORM (with policy grouping), drizzle adapter (native PG/SQLite with parameter-aware batching; per-row fallback for other providers), and memory adapter.
  • SDK layer: putManyImpl in executor.ts drops the old delete+createMany loop in favour of a single upsertMany call; created_at is included in values but excluded from the update list, correctly preserving original timestamps on re-writes.
  • Ergonomics: Collection-scoped putMany/removeMany helpers are added to PluginStorageCollectionFacade, removing the need for callers to manually inject the collection name.

Confidence Score: 5/5

Safe to merge; the change is well-scoped and all policy-enforcement paths are covered by the new tests.

Policy validation (both create and update) is enforced at the ORM layer before any adapter call. Empty-target and empty-update guards exist at both ORM and adapter levels. The native PG/SQLite path uses correct excluded.column references and parameter-aware batching. The ORM fallback for adapters without native upsertMany is logically sound. Cross-tenant write rejection and invalid-shape rejection are both tested. No correctness gaps were found in the changed paths.

No files require special attention. test-config.ts bypasses the ORM fallback when calling upsertMany on the lazy test DB, but this is benign since the memory adapter always provides the method.

Important Files Changed

Filename Overview
packages/core/fumadb/src/query/orm/index.ts Core ORM upsertMany implementation: applies per-row create/update policies, groups permitted rows by policy condition key, then dispatches to native adapter batch path or per-row upsert fallback. Logic is correct; conditionKey serialization is stable for deterministic policy output.
packages/core/fumadb/src/adapters/drizzle/query.ts Adds native INSERT … ON CONFLICT DO UPDATE for PostgreSQL/SQLite with parameter-aware batch sizing, plus a per-row upsert fallback for other providers. Helper countConditionParameters correctly handles all Condition variants for batch-size arithmetic.
packages/core/fumadb/src/adapters/memory/index.ts Memory adapter upsertMany: validates target/update are non-empty, iterates values, matches existing row by both v.where and target column equality, patches or creates. Correctly handles previous vacuous-truth concern by guarding target.length.
packages/core/sdk/src/executor.ts Routes plugin-storage putMany through upsertMany instead of delete+createMany. created_at is correctly included in values (for new rows) but excluded from the update list, preserving original timestamps on re-writes. Adds collection-scoped putMany/removeMany helpers.
packages/core/fumadb/src/query/index.ts Adds upsertMany to the AbstractQuery interface with correct generic typing over table/column names. No where clause exposed publicly — policy constraints are handled internally by the ORM layer.
packages/core/fumadb/src/query/table-policy.test.ts Adds focused policy tests: bulk upsert that spans tenants fails fast; invalid target/update shapes throw; intra-tenant bulk upsert correctly updates existing and inserts new rows.
packages/core/sdk/src/plugin-storage.ts Adds PluginStorageCollectionPutManyInput, PluginStorageCollectionRemoveManyInput interfaces and extends PluginStorageCollectionFacade with typed putMany/removeMany. Types are consistent with the executor implementation.
packages/core/sdk/src/plugin-storage.test.ts Updates test plugin to use the new collection-scoped ctx.storage.toolCalls.putMany/removeMany API, removing the manual collection-name mapping. Cleaner ergonomics with no logic change.
packages/core/sdk/src/fuma-runtime.ts Threads upsertMany through makeSafeFumaQuery by delegating to db.upsertMany — consistent with how other operations (upsert, updateMany) are wired.
packages/core/sdk/src/test-config.ts Adds upsertMany to the lazy test DB shim with a guard against adapters that don't implement it at the adapter level. The guard bypasses the ORM fallback path, but is benign in practice since the memory adapter always provides the method.
packages/plugins/openapi/src/sdk/store.test.ts Adds stub putMany/removeMany implementations (Effect.void) to the OpenAPI store mock to satisfy the updated PluginStorageCollectionFacade interface.
.changeset/plugin-storage-bulk-upserts.md Patch changeset for both fumadb and sdk packages; description accurately summarises the change.

Reviews (2): Last reviewed commit: "fix(fumadb): validate bulk upsert confli..." | Re-trigger Greptile

Comment thread packages/core/fumadb/src/adapters/memory/index.ts
Comment thread packages/core/fumadb/src/adapters/drizzle/query.ts Outdated
aryasaatvik added a commit to aryasaatvik/executor that referenced this pull request Jun 24, 2026
## Summary

- Mirror the upstream bulk-upsert validation hardening from
UsefulSoftwareCo#1098.
- Reject empty `upsertMany` conflict targets at the public FumaDB query
boundary.
- Add adapter-level empty-target guards for Drizzle and memory adapters.
- Add a table-policy regression test for invalid bulk upsert
target/update shapes.

## Validation

- `bunx vitest run src/query/table-policy.test.ts` from
`packages/core/fumadb`
- `bunx oxlint --no-ignore --deny-warnings src/query/orm/index.ts
src/adapters/drizzle/query.ts src/adapters/memory/index.ts
src/query/table-policy.test.ts` from `packages/core/fumadb`
- `git diff --check`

## Notes

`bun run --cwd packages/core/fumadb typecheck` is currently blocked in
this checkout by the existing missing `@libsql/client` import in
`src/adapters/drizzle/runtime-ensure.test.ts`. Root oxlint intentionally
ignores `packages/core/fumadb/`, so the file-level lint was run from the
package with `--no-ignore`.
aryasaatvik added a commit to aryasaatvik/executor that referenced this pull request Aug 18, 2026
## Summary

- Mirror the upstream bulk-upsert validation hardening from
UsefulSoftwareCo#1098.
- Reject empty `upsertMany` conflict targets at the public FumaDB query
boundary.
- Add adapter-level empty-target guards for Drizzle and memory adapters.
- Add a table-policy regression test for invalid bulk upsert
target/update shapes.

## Validation

- `bunx vitest run src/query/table-policy.test.ts` from
`packages/core/fumadb`
- `bunx oxlint --no-ignore --deny-warnings src/query/orm/index.ts
src/adapters/drizzle/query.ts src/adapters/memory/index.ts
src/query/table-policy.test.ts` from `packages/core/fumadb`
- `git diff --check`

## Notes

`bun run --cwd packages/core/fumadb typecheck` is currently blocked in
this checkout by the existing missing `@libsql/client` import in
`src/adapters/drizzle/runtime-ensure.test.ts`. Root oxlint intentionally
ignores `packages/core/fumadb/`, so the file-level lint was run from the
package with `--no-ignore`.
@aryasaatvik
aryasaatvik force-pushed the contrib/plugin-storage-bulk-writes branch 3 times, most recently from 788dd67 to d34c58b Compare August 27, 2026 16:44
@aryasaatvik
aryasaatvik force-pushed the contrib/plugin-storage-bulk-writes branch from 6da0c26 to aa9c25e Compare August 28, 2026 06:50
The collection-level putMany/removeMany the branch added duplicate no
production consumer; both call sites use the facade. Drop them and keep
the branch to its real contribution, the atomic upsert implementation.

Add a case proving a mid-batch failure leaves pre-existing rows intact,
and make the fault injector unconditional so removing the transaction
fails the test on the data rather than silently disarming it.
@RhysSullivan
RhysSullivan merged commit 02b52cd into UsefulSoftwareCo:main Aug 28, 2026
40 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants