Add MariaDB backend#873
Open
phdoerfler wants to merge 11 commits into
Open
Conversation
phdoerfler
marked this pull request as ready for review
July 14, 2026 19:57
Latest tag is v0.29.0; the SQLite backend work also changes the SqlMappingLike dialect API (new abstract members, defaultOffsetForSubquery renamed to normalizeOffsetLimit), so the next release must start a new binary-incompatible minor line for MiMa.
Datetime tests in the MSSQL suite (MovieSuite's `moviesShownBetween` and computed-field cases) fail on a developer machine in a non-UTC time zone, though they pass in CI. The MSSQL test codec encodes an OffsetDateTime argument via a zone-naive `java.sql.Timestamp`, and mssql-jdbc binds it into a DATETIMEOFFSET parameter using the ambient JVM time zone. On a UTC+2 host the filter bound shifts by two hours, admitting an extra row. Postgres is immune (native `Meta[OffsetDateTime]`); Oracle uses the same codec but ojdbc binds it without the shift; CI runs in UTC so it never surfaces there.
A new Doobie-based SQL backend for SQLite, mirroring the doobie-oracle/
doobie-mssql module shape: a single dialect mapping supplying the
SqlMappingLike dialect hooks, plus a test harness wiring up the shared
sql-core suites against a fresh temp-file SQLite database per test
suite (no docker service needed, since SQLite runs in-process).
Two dialect-specific choices, documented inline: LIMIT/OFFSET uses
SQLite's legacy comma form (`LIMIT offset, limit`), since the shared
query builder's fixed offset-then-limit render order is incompatible
with SQLite's OFFSET-must-follow-LIMIT grammar; and LIKE
case-sensitivity relies on a connection-level `case_sensitive_like`
pragma paired with `UPPER()`-normalization for case-insensitive matches.
SQLite has two grammar gaps no pre-existing dialect hook could route
around, addressed in sql-core with existing dialects' behavior
unchanged:
- No correlated FROM-clause subqueries (no LATERAL equivalent).
`SqlSelect.nest()` embeds a parent-constraint equality predicate into
a nested subquery's own WHERE clause - redundant with the enclosing
JOIN ... ON clause, and only useful as an optimization for genuinely
lateral-evaluated subqueries (Postgres/Oracle LATERAL, MSSQL
CROSS/OUTER APPLY). On a dialect with no lateral join at all, that
predicate references a column out of scope and fails at runtime with
"no such column". A derived capability, `supportsLateralJoin` (false
exactly when `mkLateral` has no lateral form to render and answers
`NotLateral`, as SQLite's does), makes `addFilterOrderByOffsetLimit`
skip embedding it.
- No parenthesized UNION branches. `SqlUnion.toFragment` used to wrap
each UNION ALL branch in parentheses unconditionally; SQLite's
compound-select grammar forbids that. Branch rendering is now a
dialect hook, `unionBranchToFragment`, which the existing dialects
implement with the previous parenthesizing behavior and SQLite
implements as the identity - safe for any number of branches since
grackle only ever uses UNION ALL, a purely associative bag union.
`DoobieSqliteMappingLike.encapsulateUnionBranch` is also extended to
wrap branches carrying their own offset/limit (not just order) into
a derived-table subquery, since a branch-level LIMIT is rejected by
SQLite regardless of parenthesization.
Fixing the LATERAL gap exposed a real correctness bug, also fixed
here: `addFilterOrderByOffsetLimit` has two "Case 1" fast paths that
apply LIMIT/OFFSET directly to a query built from pre-existing joins,
trusting `oneToOne && predIsOneToOne` to mean one row per key. That's
only guaranteed when a lateral-evaluated subquery produced those joins
(at most one contribution per outer row); a join chain containing a
nested one-to-many hop (e.g. a windowed/limited grandchild list) can
produce both a real match row and a NULL-extended LEFT JOIN row
sharing the same key, and an ORDER BY/LIMIT on top with no
de-duplication picks whichever the database's tie-breaking favors.
Both fast paths are now gated behind `supportsLateralJoin`, safe to
skip only when the dialect genuinely evaluates subqueries laterally,
or when there's provably no risk (no joins, or no offset/limit at all).
The offsetDateTime test codec normalizes to UTC on decode: SQLite has
no native timestamptz, so it stores and returns whatever literal
offset is written in the seed data, whereas Postgres/Oracle/MSSQL's
drivers hand back a UTC-normalized value regardless of storage;
without normalizing, otherwise-correct results failed pure
string-equality against the shared UTC ("Z")-formatted expected-JSON
fixtures.
Also applies the shared utcTestSettings (pinning the forked test JVM to
UTC) to doobiesqlite for consistency with the other backends, even
though SQLite's own datetime handling is UTC-normalized independent of
host time zone.
munit-cats-effect's 30s default is not enough headroom for the database-backed suites on a constrained machine: sbt runs each backend's tests in its own forked JVM, several of which contend for one docker daemon, and under that load individual tests have been observed to exceed 30s and fail with spurious TimeoutExceptions unrelated to what they test. Introduce a shared SqlDatabaseSuite base trait in sql-core's test scope, extended by DoobieDatabaseSuite and SqlPgDatabaseSuite (a single base rather than per-trait overrides, since DoobiePgDatabaseSuite mixes in both and separate overrides of munitIOTimeout would conflict), raising the timeout to 2 minutes for every backend suite.
defaultOffsetForSubquery supplied dialect-required offset/limit defaults only for selects rendered as subqueries, leaving the true query root unnormalized. That gap was invisible for Postgres and Oracle (identity implementations) and harmless for MSSQL (ORDER BY without OFFSET is legal at the root, unlike inside a derived table), but fatal for SQLite's comma-form LIMIT clause: a flat root query with an offset and no limit - the one shape whose offset survives to the top level unwrapped - rendered a dangling "LIMIT n," and failed at runtime with an incomplete-input error. Rename the hook to normalizeOffsetLimit and apply it once in SqlSelect.toFragment, so every rendered select is normalized wherever it appears; SubqueryRef no longer applies it separately. The only rendering change for existing dialects is that MSSQL root-level ordered queries now carry a redundant-but-legal "OFFSET 0 ROWS". Adds a shared regression test covering the flat root-offset shape, run by all backends.
SQLite sorts NULLs low by default (first in ASC, last in DESC), but orderToFragment carried the Postgres conditional, which only emits an explicit NULLS FIRST/LAST clause on the cases that deviate from a nulls-high default. On SQLite that renders the explicit clause exactly where the engine default already matches and stays silent on the two cases that need correcting, so orderings requesting the non-default placement (ascending nulls-last, descending nulls-first) put NULLs at the wrong end. Emit the clause on the mirror-image cases instead - the same polarity correction the MSSQL dialect makes, expressed with SQLite's native syntax. No existing fixture could observe this: grackle core re-sorts fetched rows in memory (stripCompiled preserves OrderBy), so the SQL-side placement only matters when it decides which rows survive a LIMIT cut, and no shared fixture orders a NULL-containing nullable column. The new NullOrderingSuite pins all four ascending/nullsLast combinations through a limit for exactly that reason.
phdoerfler
force-pushed
the
topic/mariadb-backend
branch
from
July 17, 2026 13:53
37bd31f to
30f6fbf
Compare
Superseded by the shared SqlNullOrderingMapping/SqlNullOrderingSuite in sql-core, landing via typelevel#870. This branch will pick that up once it rebases past typelevel#870's merge.
Superseded by the shared SqlNullOrderingMapping/SqlNullOrderingSuite in sql-core, landing via typelevel#870. This branch will pick that up once it rebases past typelevel#870's merge.
Superseded by the shared SqlNullOrderingMapping/SqlNullOrderingSuite in sql-core, landing via typelevel#870. This branch will pick that up once it rebases past typelevel#870's merge.
phdoerfler
force-pushed
the
topic/mariadb-backend
branch
from
July 17, 2026 20:52
30f6fbf to
9e5e622
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds MariaDB as a backend. Again, I let the LLM do the most of the work similar to the other recent backend additions (SQLite #864, H2 #866, MySQL #872): The LLM was instructed to do TDD and perform a critical review and I looked over the mapping code as well. Depends on #872 (MySQL) — it's built as a close fork of that module rather than a from-scratch port, since MariaDB and MySQL share almost their entire dialect surface. It also reuses the test data from the MySQL branch.
The biggest difference to MySQL: MariaDB has no
LATERALderived tables in any released version (unlike MySQL, which has had them natively since 8.0.14). So this backend uses the sameNotLateralmachinery SQLite/H2 use, not MySQL's native-LATERAL rendering — which is also why it has to sit on top of the SQLite PR'sSqlMappingLikechanges rather than being independent of that work.Then there's the bit with the driver license:
mariadb-java-clientis LGPL 2.1 — a different (and more permissive) license than the MySQL backend'smysql-connector-j(GPLv2 + Universal FOSS Exception). Used the vendor's own connector rather than reusing the MySQL driver via protocol compatibility, since it's the correctly-supported choice and the license is friendlier anyway.Summary
grackle-doobie-mariadb: forked from the already-complete MySQL backend by copying its source tree and renaming, then applying exactly one structural dialect change (mkLateral→NotLateral, see above). Everything else in the dialect started as a byte-identical copy of the MySQL version and was corrected via TDD against a live container wherever a shared-suite test disagreed.In practice, only one thing needed correcting: MariaDB's
JSONtype is aLONGTEXTalias, not a native CAST target the way MySQL'sJSONtype is —CAST(NULL AS JSON)is invalid SQL on MariaDB (ERROR 1064). That case now falls through to a bareNULLinstead (same treatment already given toFLOAT/DOUBLE, since an ascribed NULL is only ever an inference hint, not semantically required). This wasn't caught by the shared test suite itself — none of the union/interface suites happen to route a nullable JSON field through that specific ascription path — so it was found by directly probingCAST(NULL AS JSON)against a live MariaDB container during the final review pass, not by a failing test. Everything else needed zero changes: the full 249-test shared suite (same suite class list every other backend wires) is green on both Scala 2.13 and 3.3.8 with that one fix.Test data — no duplication
The
mariadbdocker-compose service reuses./testdata/mysql/unchanged, via a second bind mount — there's no separatetestdata/mariadb/directory. Verified this actually round-trips correctly (not just assumed dialect-compatible): checked UTF-8 fidelity of the fixture's accented city/country names through the mounted MariaDB container directly, byte-for-byte identical to how they load into MySQL. MariaDB's client didn't turn out to have MySQL's latin1-handshake charset bug, so theentrypoint.shcharset-pin workaround the MySQL backend needs wasn't necessary here.Version floor
Docker service defaults to
mariadb:${MARIADB_VERSION:-lts}— the official image'sltstag always tracks MariaDB's current LTS release (currently 12.3, under MariaDB's newer release policy where the.3release of each major version is the long-term-support one), so there's no version number to keep re-pinning. Unlike the MySQL backend, there's no independent older-version probe here — no single MariaDB feature analogous to MySQL's "LATERAL since 8.0.14" forces a particular floor, so the floor is simply "developed and tested against current LTS."