feat(cdc): native SQLite CDC source (db.cdc.sqlite)

[skhaz]

Why

The runtime streams Postgres row changes via db.cdc.postgres (#323) but had no SQLite equivalent. SQLite has no logical-replication slot, so the native mechanism is the preupdate hook, which observes row changes on the connection the runtime writes through.

This is live, in-process observation — not durable CDC. Changes made while the runtime is down, or by an external process writing the file, are not captured. Cursors are session-scoped {epoch, sequence}, not a replayable log position. There is zero schema intrusion: no tables are created in the application database.

What

Adds a db.cdc.sqlite registry kind backed by a supervised Source that installs SQLite preupdate + commit/rollback hooks on the target db.sql.sqlite pool's writer connection and emits insert/update/delete (plus an optional per-consumer snapshot bootstrap) through the existing, engine-agnostic cdc Lua module (cdc.stream / list_sources / source).

• Single capture owner per database. Ownership is a token registry keyed by the canonical file; a second source on the same DB is refused, pre-existing hooks are never clobbered, and Update/Stop hand off ownership cleanly (release is token-guarded; a stale owner never clears the new owner's hooks on the shared single writer connection). The ConnectHook re-binds the current owner across ConnMaxLifetime reconnects.
• Per-consumer snapshot. A subscriber requesting snapshot=true gets a consistent point-in-time image of the allowed tables, then live changes, with no gaps. Consistency is guaranteed by briefly fencing on the single pool writer connection before latching a read transaction on a dedicated read-only connection; rows stream on a helper goroutine so other consumers are never stalled. snapshot/error events bypass a consumer's op filter.
• Bounded, never stalls the writer. The commit hook never blocks: transactions are handed off non-blockingly, and both queued transactions and per-transaction rows/bytes are bounded. On overflow the source faults — it drops buffered data and emits one terminal op="error" gap event to all subscribers — while the application commit always succeeds (the hook always returns 0). A laggard subscriber gets a terminal error and is dropped, never backpressuring the writer.
• Session-scoped cursor. Each Start() mints an epoch; lsn is a per-session monotonic sequence, documented as not a durable replay cursor. No wippy_cdc_offsets table, no durable checkpoint.
• Correct identity. The SQLite database name is captured (schema); writes to temp/attached databases are not mis-reported as main. Column metadata is invalidated when PRAGMA schema_version changes (e.g. ALTER TABLE), with a defensive column-count guard.

Layering: service/sql exposes a build-tagged driver seam (RegisterDriver); the SQLite pool owns the concrete driver and connection hooks, and the CDC source consumes them — it never reaches back through driver globals. api/service/cdc gains the db.cdc.sqlite kind, SQLiteConfig, and a composite inspector/streamer so the Lua module observes both engines. The SQL-engine registry refactor is included here (the CDC driver seam depends on it).

Limitation (by design)

Capture is in-process and live-only. This is the trade for zero schema intrusion and lowest overhead. Durable snapshot/restart semantics would require a runtime-managed outbox with consumer acknowledgements, or trigger/outbox tables (schema intrusion) — out of scope for v1.

Portability

Requires the sqlite_preupdate_hook build tag (a compile-time SQLite feature) and CGO (the mattn/go-sqlite3 driver). Without the tag the source fails loudly instead of silently capturing nothing. WAL mode has same-host / networked-filesystem constraints (a single machine must own the -wal/-shm); this matches how db.sql.sqlite already runs.

Setup

Build with the tag (wired into the Makefile): make build-wippy-local.

- { name: db,  kind: db.sql.sqlite, file: /path/app.db, lifecycle: { auto_start: true } }
- { name: cdc, kind: db.cdc.sqlite, db_resource: app:db, lifecycle: { auto_start: true } }

Lua:

local s = cdc.stream("app:cdc", { tables = {"users"}, ops = {"insert"}, snapshot = true })
local change = s:channel():receive()
-- { op, table, schema, before, after, source, lsn }  (op="error" carries a gap reason)

Testing

• make lint (tags race,sqlite_preupdate_hook): 0 issues. Cross-platform CI green (Linux + Windows).
• 19 new test functions + 6 benchmarks (-race), covering every blocker:
  • Snapshot: late subscriber gets a consistent image then live; op-filtered consumer still receives snapshot rows; snapshot covers multiple tables; snapshot with concurrent live writes observes every row with no gap (convergence); restart re-snapshots (no durable checkpoint).
  • Ownership: single-owner token registry (refusal + token-guarded release); second source on the same DB is refused; Stop cleans up even when the caller context is already cancelled (new source can then claim).
  • Overload: overflow faults and emits one terminal gap while the writer is not stalled; the application commit still succeeds and all rows are durably written; a subscriber joining a faulted source gets a terminal error.
  • Identity: ALTER TABLE mid-stream yields correct column names; writes to a temp database are not reported as main.
  • Fan-out: per-subscriber table filtering; control events (snapshot/error) bypass op/table filters.
  • Unit: schemaAllowed, normalizeSchema, opString, approxRowSize.
• Full-stack live E2E (TestLuaSeesRealRunningSQLiteSourceAndItsChanges): registry entry → supervisor auto-start → Source.Start → preupdate hook → drain → Lua cdc.stream → dispatcher → subscribe → channel receive, asserting op/source/schema/table/after on a real WAL SQLite file.
• Performance matrix (write-throughput + p50/p99/p999 commit latency): hooks disabled vs enabled-no-subscribers vs one subscriber vs large BLOB vs large transaction vs saturated subscriber. Capture adds ~1.5 µs p50 on tiny commits; a saturated (never-draining) subscriber does not stall the writer (≈ enabled-no-subscriber), and the CDC driver adds no steady-state overhead when no source is active.

[wolfy-j]

Still in a pipeline, since it's not in a rush, i'll keep it there for some time.

[wolfy-j]

Blocking correctness findings in snapshot mode:\n\n1. Subscribe registers the subscriber for live events before fenceAndBegin establishes the snapshot. Any commit in that window is both buffered as live and included in the snapshot, so the consumer receives duplicates after the snapshot phase. TestIntegrationSnapshotWithConcurrentWritesNoGap hides this by storing IDs in a set and never asserting exactly-once delivery. The API needs an explicit barrier/cursor protocol (or documented/tested at-least-once semantics with usable dedupe identity) before claiming “consistent image, then live changes, with no gaps.”\n2. A snapshot subscription made before Source.Start silently calls finishSnapshot() and continues without any snapshot. The manager exposes the source before asynchronous supervisor startup/update completes, so this is reachable, not theoretical. It must wait/fail explicitly; silently degrading semantics is not acceptable.\n\nThe preupdate hook, bounded writer path, and ownership token work are thoughtful, but snapshot handoff is the central CDC contract and currently is not sound.