Add H2 backend#866
Open
phdoerfler wants to merge 10 commits into
Open
Conversation
phdoerfler
marked this pull request as ready for review
July 13, 2026 19:32
phdoerfler
force-pushed
the
topic/h2-backend
branch
from
July 14, 2026 13:06
8d4b22d to
3df8e48
Compare
Latest tag is v0.29.0; the SQLite backend work also changes the SqlMappingLike dialect API (new abstract members, defaultOffsetForSubquery renamed to normalizeOffsetLimit), so the next release must start a new binary-incompatible minor line for MiMa.
Datetime tests in the MSSQL suite (MovieSuite's `moviesShownBetween` and computed-field cases) fail on a developer machine in a non-UTC time zone, though they pass in CI. The MSSQL test codec encodes an OffsetDateTime argument via a zone-naive `java.sql.Timestamp`, and mssql-jdbc binds it into a DATETIMEOFFSET parameter using the ambient JVM time zone. On a UTC+2 host the filter bound shifts by two hours, admitting an extra row. Postgres is immune (native `Meta[OffsetDateTime]`); Oracle uses the same codec but ojdbc binds it without the shift; CI runs in UTC so it never surfaces there.
A new Doobie-based SQL backend for SQLite, mirroring the doobie-oracle/
doobie-mssql module shape: a single dialect mapping supplying the
SqlMappingLike dialect hooks, plus a test harness wiring up the shared
sql-core suites against a fresh temp-file SQLite database per test
suite (no docker service needed, since SQLite runs in-process).
Two dialect-specific choices, documented inline: LIMIT/OFFSET uses
SQLite's legacy comma form (`LIMIT offset, limit`), since the shared
query builder's fixed offset-then-limit render order is incompatible
with SQLite's OFFSET-must-follow-LIMIT grammar; and LIKE
case-sensitivity relies on a connection-level `case_sensitive_like`
pragma paired with `UPPER()`-normalization for case-insensitive matches.
SQLite has two grammar gaps no pre-existing dialect hook could route
around, addressed in sql-core with existing dialects' behavior
unchanged:
- No correlated FROM-clause subqueries (no LATERAL equivalent).
`SqlSelect.nest()` embeds a parent-constraint equality predicate into
a nested subquery's own WHERE clause - redundant with the enclosing
JOIN ... ON clause, and only useful as an optimization for genuinely
lateral-evaluated subqueries (Postgres/Oracle LATERAL, MSSQL
CROSS/OUTER APPLY). On a dialect with no lateral join at all, that
predicate references a column out of scope and fails at runtime with
"no such column". A derived capability, `supportsLateralJoin` (false
exactly when `mkLateral` has no lateral form to render and answers
`NotLateral`, as SQLite's does), makes `addFilterOrderByOffsetLimit`
skip embedding it.
- No parenthesized UNION branches. `SqlUnion.toFragment` used to wrap
each UNION ALL branch in parentheses unconditionally; SQLite's
compound-select grammar forbids that. Branch rendering is now a
dialect hook, `unionBranchToFragment`, which the existing dialects
implement with the previous parenthesizing behavior and SQLite
implements as the identity - safe for any number of branches since
grackle only ever uses UNION ALL, a purely associative bag union.
`DoobieSqliteMappingLike.encapsulateUnionBranch` is also extended to
wrap branches carrying their own offset/limit (not just order) into
a derived-table subquery, since a branch-level LIMIT is rejected by
SQLite regardless of parenthesization.
Fixing the LATERAL gap exposed a real correctness bug, also fixed
here: `addFilterOrderByOffsetLimit` has two "Case 1" fast paths that
apply LIMIT/OFFSET directly to a query built from pre-existing joins,
trusting `oneToOne && predIsOneToOne` to mean one row per key. That's
only guaranteed when a lateral-evaluated subquery produced those joins
(at most one contribution per outer row); a join chain containing a
nested one-to-many hop (e.g. a windowed/limited grandchild list) can
produce both a real match row and a NULL-extended LEFT JOIN row
sharing the same key, and an ORDER BY/LIMIT on top with no
de-duplication picks whichever the database's tie-breaking favors.
Both fast paths are now gated behind `supportsLateralJoin`, safe to
skip only when the dialect genuinely evaluates subqueries laterally,
or when there's provably no risk (no joins, or no offset/limit at all).
The offsetDateTime test codec normalizes to UTC on decode: SQLite has
no native timestamptz, so it stores and returns whatever literal
offset is written in the seed data, whereas Postgres/Oracle/MSSQL's
drivers hand back a UTC-normalized value regardless of storage;
without normalizing, otherwise-correct results failed pure
string-equality against the shared UTC ("Z")-formatted expected-JSON
fixtures.
Also applies the shared utcTestSettings (pinning the forked test JVM to
UTC) to doobiesqlite for consistency with the other backends, even
though SQLite's own datetime handling is UTC-normalized independent of
host time zone.
munit-cats-effect's 30s default is not enough headroom for the database-backed suites on a constrained machine: sbt runs each backend's tests in its own forked JVM, several of which contend for one docker daemon, and under that load individual tests have been observed to exceed 30s and fail with spurious TimeoutExceptions unrelated to what they test. Introduce a shared SqlDatabaseSuite base trait in sql-core's test scope, extended by DoobieDatabaseSuite and SqlPgDatabaseSuite (a single base rather than per-trait overrides, since DoobiePgDatabaseSuite mixes in both and separate overrides of munitIOTimeout would conflict), raising the timeout to 2 minutes for every backend suite.
defaultOffsetForSubquery supplied dialect-required offset/limit defaults only for selects rendered as subqueries, leaving the true query root unnormalized. That gap was invisible for Postgres and Oracle (identity implementations) and harmless for MSSQL (ORDER BY without OFFSET is legal at the root, unlike inside a derived table), but fatal for SQLite's comma-form LIMIT clause: a flat root query with an offset and no limit - the one shape whose offset survives to the top level unwrapped - rendered a dangling "LIMIT n," and failed at runtime with an incomplete-input error. Rename the hook to normalizeOffsetLimit and apply it once in SqlSelect.toFragment, so every rendered select is normalized wherever it appears; SubqueryRef no longer applies it separately. The only rendering change for existing dialects is that MSSQL root-level ordered queries now carry a redundant-but-legal "OFFSET 0 ROWS". Adds a shared regression test covering the flat root-offset shape, run by all backends.
SQLite sorts NULLs low by default (first in ASC, last in DESC), but orderToFragment carried the Postgres conditional, which only emits an explicit NULLS FIRST/LAST clause on the cases that deviate from a nulls-high default. On SQLite that renders the explicit clause exactly where the engine default already matches and stays silent on the two cases that need correcting, so orderings requesting the non-default placement (ascending nulls-last, descending nulls-first) put NULLs at the wrong end. Emit the clause on the mirror-image cases instead - the same polarity correction the MSSQL dialect makes, expressed with SQLite's native syntax. No existing fixture could observe this: grackle core re-sorts fetched rows in memory (stripCompiled preserves OrderBy), so the SQL-side placement only matters when it decides which rows survive a LIMIT cut, and no shared fixture orders a NULL-containing nullable column. The new NullOrderingSuite pins all four ascending/nullsLast combinations through a limit for exactly that reason.
phdoerfler
force-pushed
the
topic/h2-backend
branch
from
July 17, 2026 13:46
3df8e48 to
6228175
Compare
Superseded by the shared SqlNullOrderingMapping/SqlNullOrderingSuite in sql-core, landing via typelevel#870. This branch will pick that up once it rebases past typelevel#870's merge.
Port of the doobie-sqlite NullOrderingSuite: all four ascending x nullsLast combinations, ordered through a limit over a fixture whose nullable column actually contains NULLs. The limit matters because grackle core re-sorts fetched rows in memory (stripCompiled preserves OrderBy), so a dialect's SQL-side NULL placement is only observable when it decides which rows survive the cut. Verified to fail (the two non-default placement cases) against the pg-polarity rendering this dialect shipped with before review, and to pass with the corrected mirror-image rendering.
Superseded by the shared SqlNullOrderingMapping/SqlNullOrderingSuite in sql-core, landing via typelevel#870. This branch will pick that up once it rebases past typelevel#870's merge.
phdoerfler
force-pushed
the
topic/h2-backend
branch
from
July 17, 2026 20:47
6228175 to
d5b47b6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR includes commits of #864. It was branched off the sqlite branch because it, too, required the lateral joins workaround. I would thus recommend merging the sqlite PR prior to this one.
The H2 implementation, too, was done assisted by Claude using the superpowers plugin, making sure that any hypothesis was treated as such and methodically proven or disproven by first writing failing tests and then implementing a fix for them. This has uncovered (and subsequently fixed) an interesting bug related to null sorting and row offsets.