Skip to content

Add MariaDB backend#873

Open
phdoerfler wants to merge 11 commits into
typelevel:mainfrom
phdoerfler:topic/mariadb-backend
Open

Add MariaDB backend#873
phdoerfler wants to merge 11 commits into
typelevel:mainfrom
phdoerfler:topic/mariadb-backend

Conversation

@phdoerfler

@phdoerfler phdoerfler commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

This adds MariaDB as a backend. Again, I let the LLM do the most of the work similar to the other recent backend additions (SQLite #864, H2 #866, MySQL #872): The LLM was instructed to do TDD and perform a critical review and I looked over the mapping code as well. Depends on #872 (MySQL) — it's built as a close fork of that module rather than a from-scratch port, since MariaDB and MySQL share almost their entire dialect surface. It also reuses the test data from the MySQL branch.

The biggest difference to MySQL: MariaDB has no LATERAL derived tables in any released version (unlike MySQL, which has had them natively since 8.0.14). So this backend uses the same NotLateral machinery SQLite/H2 use, not MySQL's native-LATERAL rendering — which is also why it has to sit on top of the SQLite PR's SqlMappingLike changes rather than being independent of that work.

Then there's the bit with the driver license:
mariadb-java-client is LGPL 2.1 — a different (and more permissive) license than the MySQL backend's mysql-connector-j (GPLv2 + Universal FOSS Exception). Used the vendor's own connector rather than reusing the MySQL driver via protocol compatibility, since it's the correctly-supported choice and the license is friendlier anyway.


Summary

grackle-doobie-mariadb: forked from the already-complete MySQL backend by copying its source tree and renaming, then applying exactly one structural dialect change (mkLateralNotLateral, see above). Everything else in the dialect started as a byte-identical copy of the MySQL version and was corrected via TDD against a live container wherever a shared-suite test disagreed.

In practice, only one thing needed correcting: MariaDB's JSON type is a LONGTEXT alias, not a native CAST target the way MySQL's JSON type is — CAST(NULL AS JSON) is invalid SQL on MariaDB (ERROR 1064). That case now falls through to a bare NULL instead (same treatment already given to FLOAT/DOUBLE, since an ascribed NULL is only ever an inference hint, not semantically required). This wasn't caught by the shared test suite itself — none of the union/interface suites happen to route a nullable JSON field through that specific ascription path — so it was found by directly probing CAST(NULL AS JSON) against a live MariaDB container during the final review pass, not by a failing test. Everything else needed zero changes: the full 249-test shared suite (same suite class list every other backend wires) is green on both Scala 2.13 and 3.3.8 with that one fix.

Test data — no duplication

The mariadb docker-compose service reuses ./testdata/mysql/ unchanged, via a second bind mount — there's no separate testdata/mariadb/ directory. Verified this actually round-trips correctly (not just assumed dialect-compatible): checked UTF-8 fidelity of the fixture's accented city/country names through the mounted MariaDB container directly, byte-for-byte identical to how they load into MySQL. MariaDB's client didn't turn out to have MySQL's latin1-handshake charset bug, so the entrypoint.sh charset-pin workaround the MySQL backend needs wasn't necessary here.

Version floor

Docker service defaults to mariadb:${MARIADB_VERSION:-lts} — the official image's lts tag always tracks MariaDB's current LTS release (currently 12.3, under MariaDB's newer release policy where the .3 release of each major version is the long-term-support one), so there's no version number to keep re-pinning. Unlike the MySQL backend, there's no independent older-version probe here — no single MariaDB feature analogous to MySQL's "LATERAL since 8.0.14" forces a particular floor, so the floor is simply "developed and tested against current LTS."

@phdoerfler
phdoerfler marked this pull request as ready for review July 14, 2026 19:57
Latest tag is v0.29.0; the SQLite backend work also changes the
SqlMappingLike dialect API (new abstract members, defaultOffsetForSubquery
renamed to normalizeOffsetLimit), so the next release must start a new
binary-incompatible minor line for MiMa.
Datetime tests in the MSSQL suite (MovieSuite's `moviesShownBetween` and
computed-field cases) fail on a developer machine in a non-UTC time zone,
though they pass in CI. The MSSQL test codec encodes an OffsetDateTime
argument via a zone-naive `java.sql.Timestamp`, and mssql-jdbc binds it
into a DATETIMEOFFSET parameter using the ambient JVM time zone. On a
UTC+2 host the filter bound shifts by two hours, admitting an extra row.
Postgres is immune (native `Meta[OffsetDateTime]`); Oracle uses the same
codec but ojdbc binds it without the shift; CI runs in UTC so it never
surfaces there.
A new Doobie-based SQL backend for SQLite, mirroring the doobie-oracle/
doobie-mssql module shape: a single dialect mapping supplying the
SqlMappingLike dialect hooks, plus a test harness wiring up the shared
sql-core suites against a fresh temp-file SQLite database per test
suite (no docker service needed, since SQLite runs in-process).

Two dialect-specific choices, documented inline: LIMIT/OFFSET uses
SQLite's legacy comma form (`LIMIT offset, limit`), since the shared
query builder's fixed offset-then-limit render order is incompatible
with SQLite's OFFSET-must-follow-LIMIT grammar; and LIKE
case-sensitivity relies on a connection-level `case_sensitive_like`
pragma paired with `UPPER()`-normalization for case-insensitive matches.

SQLite has two grammar gaps no pre-existing dialect hook could route
around, addressed in sql-core with existing dialects' behavior
unchanged:

- No correlated FROM-clause subqueries (no LATERAL equivalent).
  `SqlSelect.nest()` embeds a parent-constraint equality predicate into
  a nested subquery's own WHERE clause - redundant with the enclosing
  JOIN ... ON clause, and only useful as an optimization for genuinely
  lateral-evaluated subqueries (Postgres/Oracle LATERAL, MSSQL
  CROSS/OUTER APPLY). On a dialect with no lateral join at all, that
  predicate references a column out of scope and fails at runtime with
  "no such column". A derived capability, `supportsLateralJoin` (false
  exactly when `mkLateral` has no lateral form to render and answers
  `NotLateral`, as SQLite's does), makes `addFilterOrderByOffsetLimit`
  skip embedding it.

- No parenthesized UNION branches. `SqlUnion.toFragment` used to wrap
  each UNION ALL branch in parentheses unconditionally; SQLite's
  compound-select grammar forbids that. Branch rendering is now a
  dialect hook, `unionBranchToFragment`, which the existing dialects
  implement with the previous parenthesizing behavior and SQLite
  implements as the identity - safe for any number of branches since
  grackle only ever uses UNION ALL, a purely associative bag union.
  `DoobieSqliteMappingLike.encapsulateUnionBranch` is also extended to
  wrap branches carrying their own offset/limit (not just order) into
  a derived-table subquery, since a branch-level LIMIT is rejected by
  SQLite regardless of parenthesization.

Fixing the LATERAL gap exposed a real correctness bug, also fixed
here: `addFilterOrderByOffsetLimit` has two "Case 1" fast paths that
apply LIMIT/OFFSET directly to a query built from pre-existing joins,
trusting `oneToOne && predIsOneToOne` to mean one row per key. That's
only guaranteed when a lateral-evaluated subquery produced those joins
(at most one contribution per outer row); a join chain containing a
nested one-to-many hop (e.g. a windowed/limited grandchild list) can
produce both a real match row and a NULL-extended LEFT JOIN row
sharing the same key, and an ORDER BY/LIMIT on top with no
de-duplication picks whichever the database's tie-breaking favors.
Both fast paths are now gated behind `supportsLateralJoin`, safe to
skip only when the dialect genuinely evaluates subqueries laterally,
or when there's provably no risk (no joins, or no offset/limit at all).

The offsetDateTime test codec normalizes to UTC on decode: SQLite has
no native timestamptz, so it stores and returns whatever literal
offset is written in the seed data, whereas Postgres/Oracle/MSSQL's
drivers hand back a UTC-normalized value regardless of storage;
without normalizing, otherwise-correct results failed pure
string-equality against the shared UTC ("Z")-formatted expected-JSON
fixtures.

Also applies the shared utcTestSettings (pinning the forked test JVM to
UTC) to doobiesqlite for consistency with the other backends, even
though SQLite's own datetime handling is UTC-normalized independent of
host time zone.
munit-cats-effect's 30s default is not enough headroom for the
database-backed suites on a constrained machine: sbt runs each
backend's tests in its own forked JVM, several of which contend for
one docker daemon, and under that load individual tests have been
observed to exceed 30s and fail with spurious TimeoutExceptions
unrelated to what they test.

Introduce a shared SqlDatabaseSuite base trait in sql-core's test
scope, extended by DoobieDatabaseSuite and SqlPgDatabaseSuite (a
single base rather than per-trait overrides, since
DoobiePgDatabaseSuite mixes in both and separate overrides of
munitIOTimeout would conflict), raising the timeout to 2 minutes for
every backend suite.
defaultOffsetForSubquery supplied dialect-required offset/limit
defaults only for selects rendered as subqueries, leaving the true
query root unnormalized. That gap was invisible for Postgres and
Oracle (identity implementations) and harmless for MSSQL (ORDER BY
without OFFSET is legal at the root, unlike inside a derived table),
but fatal for SQLite's comma-form LIMIT clause: a flat root query with
an offset and no limit - the one shape whose offset survives to the
top level unwrapped - rendered a dangling "LIMIT n," and failed at
runtime with an incomplete-input error.

Rename the hook to normalizeOffsetLimit and apply it once in
SqlSelect.toFragment, so every rendered select is normalized wherever
it appears; SubqueryRef no longer applies it separately. The only
rendering change for existing dialects is that MSSQL root-level
ordered queries now carry a redundant-but-legal "OFFSET 0 ROWS".

Adds a shared regression test covering the flat root-offset shape,
run by all backends.
SQLite sorts NULLs low by default (first in ASC, last in DESC), but
orderToFragment carried the Postgres conditional, which only emits an
explicit NULLS FIRST/LAST clause on the cases that deviate from a
nulls-high default. On SQLite that renders the explicit clause exactly
where the engine default already matches and stays silent on the two
cases that need correcting, so orderings requesting the non-default
placement (ascending nulls-last, descending nulls-first) put NULLs at
the wrong end. Emit the clause on the mirror-image cases instead - the
same polarity correction the MSSQL dialect makes, expressed with
SQLite's native syntax.

No existing fixture could observe this: grackle core re-sorts fetched
rows in memory (stripCompiled preserves OrderBy), so the SQL-side
placement only matters when it decides which rows survive a LIMIT cut,
and no shared fixture orders a NULL-containing nullable column. The new
NullOrderingSuite pins all four ascending/nullsLast combinations
through a limit for exactly that reason.
@phdoerfler
phdoerfler force-pushed the topic/mariadb-backend branch from 37bd31f to 30f6fbf Compare July 17, 2026 13:53
Superseded by the shared SqlNullOrderingMapping/SqlNullOrderingSuite
in sql-core, landing via typelevel#870. This branch will pick
that up once it rebases past typelevel#870's merge.
Superseded by the shared SqlNullOrderingMapping/SqlNullOrderingSuite
in sql-core, landing via typelevel#870. This branch will pick
that up once it rebases past typelevel#870's merge.
Superseded by the shared SqlNullOrderingMapping/SqlNullOrderingSuite
in sql-core, landing via typelevel#870. This branch will pick
that up once it rebases past typelevel#870's merge.
@phdoerfler
phdoerfler force-pushed the topic/mariadb-backend branch from 30f6fbf to 9e5e622 Compare July 17, 2026 20:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant