Skip to content

Fix SQLite variable limit in exports; extract shared-project creation flow - #453

Merged
tomcardoso merged 1 commit into
mainfrom
fix/issue-433-442-batch
Jul 17, 2026
Merged

Fix SQLite variable limit in exports; extract shared-project creation flow#453
tomcardoso merged 1 commit into
mainfrom
fix/issue-433-442-batch

Conversation

@tomcardoso

Copy link
Copy Markdown
Owner

Summary

#433 — Exports and shared-project conversion fail with >999 contacts (SQLite variable limit)

export:project, export:all-contacts, and export:vcard-all-contacts built unchunked IN (...) clauses from contact-ID lists that could exceed SQLite's default SQLITE_LIMIT_VARIABLE_NUMBER of 999 — both from the full member/contact-ID set used for the bulk email/phone/link/handle fetches, and from the contactIds selection filter used for "export selected" flows.

  • Bulk sub-table fetches now go through a new fetchContactSubTables() helper that chunks IN clauses at 500 (mirroring the pattern pushAppendOnly already uses in the sync engine), via a small generic chunkedIn() helper.
  • The contactIds selection filter (which can be arbitrarily large — a user can select-all on a huge table) now filters in JS with a Set instead of building a SQL IN clause, since its size isn't bounded by anything the code controls.
  • projects:convertToShared / regenerateShared's part of this issue (an UPDATE ... WHERE id IN (...) over member contact IDs) turned out to already be fixed as a side effect of the Sync: explicit replication state — per-project push tracking, full-scan pull, deletion tombstones #449 sync-replication-state redesign — it's now a single project-scoped DELETE FROM sync_pushed WHERE project_id = ? with no per-contact IN clause at all.

#442 — Extract duplicated flows: shared-project creation, export bulk-fetch, sync-engine statement reuse

Doing together with #433 since the issue calls out they touch the same lines.

  • Item 1 (shared-project creation): projects:createShared, convertToShared, and regenerateShared each repeated the same ~40-line "create shared DB file → write project metadata → write local project record → compensating cleanup on failure" flow, and had drifted: regenerateShared was missing the compensating cleanup (evict the shared DB connection + unlink the file) that the other two had, so a failure in its local DB write left an orphaned shared file on disk. Extracted into one createSharedProjectFile() helper — the missing cleanup comes free.
  • Item 2 (export bulk-fetch): the same fetchContactSubTables() helper above roughly halves the duplicated per-sub-table fetch/group pattern (previously repeated 8 times across the two handlers). Also extracted the identical "write rows to a temp file, atomically rename into place" tail of both handlers into writeExportRows().
  • Item 3 (sync-engine statement reuse): left out of this PR. The issue itself frames it as low priority ("first stop if sync ever feels slow") and unrelated to the Exports and shared-project conversion fail with >999 contacts (SQLite variable limit) #433 chunking work — didn't want to touch the sync engine without a concrete need. Can file a separate issue if it's still wanted.

Closes #433
Closes #442

Test plan

  • npm run rebuild:node && npm test — full suite passes (519 tests)
  • New src/test/export.test.ts: exports 1200 contacts / 1200 project members without hitting "too many SQL variables" (would have thrown pre-fix); exports a filtered selection of >999 contact IDs correctly (excludes non-selected rows); sub-table data (emails/phones/handles) still round-trips correctly through the refactored fetch path
  • New tests in src/test/projects.test.ts for createShared/convertToShared/regenerateShared: happy-path behavior (payload returned, sync_pushed cleared, is_shared flag set), and a regression test proving regenerateShared now cleans up the shared file when the local DB write fails (previously missing — the mocked closeSharedDb/unlinkSync calls would not have happened on the old code)
  • npx tsc --noEmit clean on both tsconfig.node.json and tsconfig.web.json
  • npx eslint . clean (only pre-existing, unrelated react-hooks/exhaustive-deps warnings)
  • npm run rebuild — restored Electron-targeted native binaries after testing

🤖 Generated with Claude Code

… flow

- export:project, export:all-contacts, and export:vcard-all-contacts built
  unchunked `IN (...)` clauses from contact-ID lists that could exceed
  SQLite's 999-variable limit — both from the full member/contact set and
  from the `contactIds` selection filter. Bulk sub-table fetches now go
  through a chunked `fetchContactSubTables()` helper (chunks at 500, mirrors
  the pattern already used in the sync engine), and filterIds now filters in
  JS via a Set instead of building a SQL IN clause, since selection size is
  user-controlled and can't be bounded (fixes #433).
- Deduplicated the export handlers' sub-table bulk-fetch/group and
  write-to-spreadsheet logic into `fetchContactSubTables()` +
  `writeExportRows()`, used by both export:project and export:all-contacts.
- Extracted the ~40-line "create shared DB file, write project metadata,
  write local project record, compensating cleanup on failure" flow shared
  by projects:createShared, convertToShared, and regenerateShared into one
  `createSharedProjectFile()` helper. This also fixes a real bug:
  regenerateShared was missing the compensating cleanup (evict shared DB
  connection + unlink the file) that the other two handlers had, so a
  failure in its local DB write left an orphaned shared file on disk
  (fixes #442, items 1 and 2; item 3, sync-engine statement reuse, is left
  as explicitly low-priority per the issue text).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@tomcardoso
tomcardoso merged commit a21c2ae into main Jul 17, 2026
1 check passed
@tomcardoso
tomcardoso deleted the fix/issue-433-442-batch branch July 17, 2026 22:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant