com.github.crinklywrappr/syncopate

A claude-quality ragtime adaptor for Datalevin.

Syncopate lets you version and evolve a Datalevin schema (and the data around it) with ragtime's battle-tested migrate/rollback machinery — while fixing the sharp edges that the SQL adaptors and hand-rolled Datalevin adaptors trip over.

Why another adaptor

It began as a toy adaptor that stored migration records as datoms, sorted history by wall-clock, and swallowed every error. That works in a demo and fails in production. Syncopate is a from-scratch redesign informed by the two blessed ragtime adaptors (ragtime.jdbc, ragtime.next-jdbc), the recurring complaints in ragtime's issue tracker, and the ideas in datalevin-surge:

• Bookkeeping stays out of your data. Applied-migration state lives in a dedicated key-value DBI on the same environment your connection already holds — never as datoms. It's invisible to d/schema and Datalog queries, and works under :closed-schema? true. This holds for both embedded connections (reusing the connection's own LMDB env) and client/server (dtlv://) connections (a KV client to the same server database, so the DBI sits beside datalevin/eav etc. on the server).
• Migrations are atomic — each direction runs inside d/with-transaction by default, with a per-migration :transaction? false escape for operations that can't (ragtime issues #107 / #119). (In client/server mode the applied-id write can't join the remote datalog transaction, so it uses the two-transaction seam — the same one :transaction? false uses; see "Notes & limitations".)
• Errors are actionable. Failures are re-thrown as ex-info carrying the migration id, direction, phase and offending step — never silently dropped (ragtime issues #130 / #146).
• Observability built in — pending and status tell you what will run before you run it (ragtime issues #85 / #127).
• Migrations are code, done right. Data transforms are referenced by fully-qualified symbol and resolved to a real var at run time — no quoting, any namespace, editor-navigable (improving on surge's quoted-form restriction, and answering ragtime issue #131).

Migration format

Migrations are EDN-first. Put one file per migration under a directory or classpath prefix; the id defaults to the filename (0001-add-users).

A migration spec is a map:

{:id           "0002-split-names"   ; optional; defaults to the filename
 :up           <step | [step ...]>
 :down         <step | [step ...]>  ; optional if :up is purely additive
 :transaction? true                 ; optional, default true
 :irreversible? false}              ; optional

A step is one of:

Step	Meaning
{:schema {attr def ...}}	add/update attributes (via update-schema)
{:schema/remove [attr ...]}	retract every datom of the attrs, then drop them
{:tx [tx-data ...]}	raw transact! data
a.namespace/fn-symbol	resolved to (fn [conn] ...) and called
an actual (fn [conn] ...)	only in .clj migration files

Sugar for elision

• :up/:down may be a single step instead of a vector.
• Omit :down and Syncopate derives it automatically — iff every :up step is a purely additive schema delta (adding attrs inverts to removing them). If :up contains a removal, a :tx, or a function, supply :down explicitly or set :irreversible? true; otherwise you get a clear build-time error.

Examples

Purely additive — :down is derived (resources/migrations/0001-add-users.edn):

{:up {:schema {:user/id   {:db/valueType :db.type/long
                           :db/unique    :db.unique/identity}
               :user/name {:db/valueType :db.type/string}}}}

Schema change plus a data transform referenced by symbol (resources/migrations/0002-split-names.edn):

{:up   [{:schema {:user/given-name  {:db/valueType :db.type/string}
                  :user/family-name {:db/valueType :db.type/string}}}
        my-app.migrations/split-user-names
        {:schema/remove [:user/name]}]
 :down [{:schema {:user/name {:db/valueType :db.type/string}}}
        my-app.migrations/join-user-names
        {:schema/remove [:user/given-name :user/family-name]}]}

A .clj migration, whose steps may be real functions (resources/migrations/0003-seed-departments.clj):

{:up   [{:schema {:dept/name {:db/valueType :db.type/string
                              :db/unique    :db.unique/identity}}}
        (fn [conn]
          (datalevin.core/transact! conn [{:dept/name "Engineering"}]))]
 :down [(fn [conn] ...)
        {:schema/remove [:dept/name]}]}

Usage

(require '[datalevin.core :as d]
         '[ragtime.core :as ragtime]
         '[syncopate.core :as syncopate])

(def conn  (d/get-conn "/path/to/db"))
(def store (syncopate/store conn))                 ; the ragtime DataStore
(def migs  (syncopate/load-resources "migrations"))

;; What's pending?
(syncopate/status store migs)
;; => {:applied [] :pending ["0001-add-users" "0002-split-names" ...]}

;; Apply everything outstanding, then roll the last one back:
(ragtime/migrate-all store {} migs)
(ragtime/rollback-last store {})

Atomic helpers (recommended — fully atomic on embedded)

ragtime.core/migrate runs the migration body and records the applied-id as two separate transactions. Syncopate's own helpers fold both writes into a single d/with-transaction, so the body and its bookkeeping commit — or roll back — as one unit (a mid-migration failure leaves neither the schema/data change nor the applied-id behind):

(syncopate/migrate! store migration)        ; atomic :up + record
(syncopate/rollback! store migration)       ; atomic :down + un-record
(syncopate/migrate-all! store migs)         ; apply all pending, each atomically (idempotent)
(syncopate/rollback-last! store migs)       ; atomically roll back the most recent

This is possible because the applied-id lives in a KV DBI on the same LMDB environment as the data. It applies to transactional migrations (the default); :transaction? false migrations fall back to the two-step path.

Mode note. The single-unit commit above is an embedded-mode property. In client/server (dtlv://) the applied-id write can't join the remote datalog transaction, so these same helpers remain the recommended entry point but fall back to the two-transaction seam (body commits first, applied-id recorded after — see Notes & limitations). Bookkeeping still lives in the KV DBI on the same server environment; only the atomic fold is lost.

syncopate/store options:

Option	Default	Meaning
:dbi-name	"__syncopate_migrations"	KV DBI holding applied-migration state

Logging

Syncopate logs through trove, a tiny facade that lets a library emit rich logs without forcing a backend on you. Calls are noop until you install a backend, so nothing is printed by default. In your app, wire trove to whatever you already use:

(require '[taoensso.trove :as trove]
         '[taoensso.trove.console])           ; or .tools-logging / .slf4j / .telemere / .timbre / .mulog

;; each integration ns exposes get-log-fn:
(trove/set-log-fn! (taoensso.trove.console/get-log-fn))
;; (trove/set-log-fn! nil)  ; noop / silence

See trove's docs for the full list of backend integrations and their options.

Every event has a stable :id, a human :msg, and structured :data (migration id, direction, step counts, elapsed ms). Levels:

Level	Events
info	:syncopate/migrate-all, :migrate-all-done, :migrated, :rolled-back, :rollback-last, :nothing-pending, :nothing-applied, :loaded
warn	:syncopate/no-migrations (nothing found at the given path/prefix)
error	:syncopate/migration-failed (carries the causing throwable and the failing step)
debug	:syncopate/migrating, :rolling-back, :step, :schema-delta, :schema-retract, :recorded, :unrecorded, :auto-down

Filtering by level/namespace/id is the backend's job (e.g. via tools.logging/slf4j config). Building :data is deferred, so disabled log levels cost effectively nothing.

Testing

Embedded (a temporary local Datalevin database):

clojure -T:build test

Client/server — the same suite run as a pure client against a real, separate datalevin server. Start one (in another terminal), then run the remote suite:

clojure -M:server                       # datalevin serv on :8898 (data/test-server)
clojure -T:build test :remote true      # connects to dtlv://…@localhost:8898

Point at an already-running / containerised server instead with clojure -T:build test :remote true :server-uri "dtlv://user:pass@host:port". ci forwards these args (clojure -T:build ci :remote true), and CI runs both modes. Each test uses a fresh database (embedded a temp dir, remote a unique dtlv://…/t<uuid> the server auto-creates), so runs are isolated.

The suite covers loading/ordering, auto-:down derivation, the reversibility guard, a full migrate/rollback round-trip (schema and data, including a symbol-referenced transform and a .clj migration), applied-id ordering, the no-schema-pollution invariant, error-context propagation, the atomic helpers, and the logging events — plus a generative suite (reduced iteration counts remote). The one exception is the atomic-rollback-on-failure guarantee (a failing transactional :up leaves neither schema change nor applied-id behind): that is embedded-only — it's the single-transaction fold client/server trades for the two-transaction seam, so it's tagged ^:embedded and skipped in the remote run.

Notes & limitations

• Prefer the atomic helpers (migrate!/rollback!/migrate-all!/ rollback-last!) — they commit the migration body and its applied-id in one transaction. The two-transaction seam (body committed, applied-id recorded after) remains only for ragtime.core/migrate and for :transaction? false migrations, which have no transaction to fold into; keep those idempotent-friendly.
• Run migrations from a single, serial writer — the normal deploy/startup path. Syncopate does not coordinate concurrent migration runs, so applying migrations from two processes (or threads) against the same store at the same time is unsupported and can corrupt the applied-migration bookkeeping.
• Auto-derived :down only covers additive schema deltas; anything that removes attributes, transacts data, or runs a function needs an explicit :down (the prior state can't be inferred).
• Reaching the connection's LMDB handle uses Datalevin internals ((.lmdb (:store @conn)) embedded; the remote store's :uri + open-kv for client/server); verified against Datalevin 1.0.0.
• Client/server (dtlv://) is supported: the store opens a KV client to the same server database (sharing the datalog connection's env). Call (syncopate/close! store) when done to release that client (no-op for embedded). Client/server migrations use the two-transaction seam (body committed, applied-id recorded after) since the KV client can't join the remote datalog transaction — keep them idempotent-friendly, as with :transaction? false.
• Datalevin needs some JVM flags (they suppress warnings / grant native access): --add-opens=java.base/java.nio=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, and --enable-native-access=ALL-UNNAMED; on JDK 24+ also --sun-misc-unsafe-memory-access=allow. The test task sets these for you; add them to your app/REPL JVM opts when embedding Syncopate.

License

Copyright © 2026 crinklywrappr.

Distributed under the Eclipse Public License 2.0