Skip to content

RQL 2.0: specification draft + TypeScript reference parser - #86

Open
kriszyp wants to merge 14 commits into
masterfrom
kris/rql-v2
Open

RQL 2.0: specification draft + TypeScript reference parser#86
kriszyp wants to merge 14 commits into
masterfrom
kris/rql-v2

Conversation

@kriszyp

@kriszyp kriszyp commented Sep 1, 2026

Copy link
Copy Markdown
Member

Summary

Resurrects this project as RQL 2.0 — a specification and reference implementation of the query-language lineage that Harper's REST syntax descends from.

Philosophy (revised during review): the spec defines the ideal language — the cleanest coherent semantics for the syntax in production use — rather than reverse-engineering any single implementation. The reference implementation optimizes for correctness and clarity for direct users; it is deliberately language-neutral in its data model so reference implementations in other languages can follow. Harper keeps its own perf-tuned parser and converges over time; its known deviations are tracked in a divergence-ledger epic (HarperFast/harper#2440) linked from Appendix D, keeping the spec vendor-independent.

  • specification/rql-2.0.md — draft spec: ABNF grammar, comparator/value semantics, element-scoped matching, the canonical parsed representation, Core/Extensions conformance profiles, v1 migration (Appendix A), compatibility aliases (Appendix B), implementation divergence ledgers (Appendix D → harper#2440), and a non-normative PostgREST dialect mapping (Appendix E).
  • src/ — zero-dependency TypeScript reference parser producing the spec's canonical model: plain serializable data ({filter, sort, select, limit, offset}), no host-framework types.
  • test/v2/ — 111 tests: the conformance suite (surface string → canonical representation), seeded from Harper's parser test suites plus new coverage.

Language highlights

  • not(...) group/scope negation — pure sugar desugared by De Morgan onto leaf not_ flags and negated element scopes; Harper support filed as HarperFast/harper#2441.

  • Small orthogonal core: 9 comparators (eq, lt, le, gt, ge, contains, starts_with, ends_with, in) with uniform not_ negation. ne/equals/not_equal/between/abbreviations are compatibility aliases with exact desugarings — one canonical form per meaning.

  • Verbatim vs. interpreted values, not strict-vs-coercive comparators: a=3 binds the string "3"; a==3 binds the number 3. The distinction lives in the value literal, so there is exactly one equality.

  • Element-scoped matching: conditions on list-valued properties match existentially, and conjunction doesn't distribute over that quantifier — so ratings=ge=3&=le=4 (one element in range) is semantically distinct from two separate conditions (any elements witness each). Chains and prop[cond&cond] sub-queries canonicalize to an ElementMatch node; between desugars into it.

  • Operator model: comparators are infix-only with an open, execution-validated name set; call syntax (select/sort/limit) is a closed, parse-validated set of result-shaping functions. (In 1.x these were one category; see Appendix A.)

  • Percent-encoding layering: paths split on literal . before per-segment decoding, so %2E expresses a literal dot in a property name.

  • PostgREST dialect mapping (Appendix E) — every PostgREST filter operator maps into Core or an extension comparator, and both dialects' logical layers are the same Group tree (prefix vs infix). Recorded as evidence the canonical model is dialect-neutral, with the real gaps (projection aliasing/casting, nulls ordering, resource embedding) and semantic deltas (embedded-filter join defaults, SQL null tri-state) named rather than papered over.

Open questions for review

  • §7 canonical serialization and §9 security bounds are TODO.
  • Whether in should also demote to an alias of or-grouped eq (kept Core for idiom and index-scan affinity).
  • Extensions profile (Appendix C) semantics are deferred to a future revision.
  • Known parser gap: list values inside nested groups ((a=in=(1,2)&b=2)) throw a syntax error — the group sub-parser lacks value-mode scanning for parenthesized lists. Fails loudly, fix planned.

Test plan

  • npm test — 111/111 passing (includes the ski-lengths element-scoping case, verbatim/interpreted distinction, %2E handling, alias desugaring equivalences)
  • Cross-check corpus against Harper's parser to populate/verify the Appendix D divergence list systematically

🤖 Generated with Claude Code

🤖 Generated with Claude Code

kriszyp and others added 14 commits December 22, 2014 12:22
Ports Harper's REST query-string parser (resources/search.ts:1221-1616)
into a standalone, reentrant TypeScript package under src/.

Key changes from the original:
- parseQuery encapsulates all mutable state (lastIndex, regex instances)
  in a per-call closure so concurrent invocations are independent.
- Query extends URLSearchParams; fast path returns URLSearchParams-backed
  Query (conditions unset), parsed path populates conditions[].
- Object.defineProperty shadows URLSearchParams.prototype.sort so a Sort
  object can be assigned on the same property name.
- group-by now has a proper break (original falls through into sort).
- 45 tests pass: all assertions from query-parse.test.js and the tier1
  REST-query-parsing block, plus reentrancy and URLSearchParams shape tests.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Grammar (ABNF draft), Core comparator/coercion/composition semantics,
canonical Query AST (extends URLSearchParams, dual-shape conditions),
conformance profiles, and the v1 migration appendix.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
v1's normalized call form for comparators (lt(price,10)) is removed in 2.0;
comparators are infix-only (open set), call syntax is the closed set of
result-shaping functions.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Language-neutral canonical model (no URLSearchParams, no dual-shape
  fast path, sort as ordered list, projection as mode+fields)
- Comparator reduction: eq/lt/le/gt/ge/contains/starts_with/ends_with/in
  with uniform not_ negation; strict-vs-coercive becomes verbatim-vs-
  interpreted value literals; ne/equals/between/aliases -> Appendix B
  compatibility desugarings
- Chaining and between desugar to plain same-path conditions
- Percent-encoding layering normative (split-then-decode; %2E works —
  verified Harper already matches, earlier report was wrong)
- prop[]= demoted to host accommodation
- New Appendix D: tracked Harper divergences

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Conjunction does not distribute over existential array matching:
skiLengths=ge=175&=le=180 (one element in range) differs from two
separate conditions (any elements witness each). Chaining and
prop[group] sub-queries canonicalize to a new ElementMatch construct;
between desugars into it. Adds the bare =op= and-continuation spelling
(new in 2.0; Harper-lineage grammar rejects it — verified). README: v2
status note.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The bare =op= spelling was motivated by a typo. Also: chain legs take a
single value (no lists), and the comparator name is required — Harper's
nameless-leg comparator inheritance is now a tracked divergence.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Drops Query extends URLSearchParams; parseQuery returns plain ParseResult.
- Parser creates fresh QP/VP regex instances per call for reentrancy.
- VP switching: VP used only after a non-eq comparator or FIQL name is
  committed, preventing premature consumption of FIQL second-= tokens.
- Chaining (&= / |=) desugars to ElementMatch{path, some:Group} with
  element-relative inner conditions (path:[] for scalar-field legs).
  Preserves existential semantics: some ONE array element satisfies all legs.
- between/not_between desugar to ElementMatch (not a flat ge+le pair).
- Bracket prop access (prop[cond]) produces ElementMatch; single-condition
  bracket normalises to plain Condition with concatenated path.
- Aliases and not_ prefix handled via resolveFiqlName at parse time.
- All 68 tests pass.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- QP tokenizer now emits &= / |= as single tokens; chain handling inside
  groups (and after verbatim = conditions) was unreachable and errored
- parseCondGroup builds ElementMatch for chains (element scoping was
  lost inside (...)/[...])
- Interpreted mode auto-converts round-trip numerals, keeping a==3
  distinct from verbatim a=3 (§5.2)
- Nameless chain legs (a=ge=1&=5) are a syntax error per the grammar
- Nested projections are records mode (brand{name} trims, not values)
- Singleton ElementMatch normalizes to a plain Condition on close
- ParseResult.filter narrowed to Group

74 tests pass (was 68).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- B1: negation scopes over the condition's own traversal (top-level ¬∃,
  in-scope predicate negation, recursive); flattening invariant exempts
  negated inner conditions; elem-cond surface (tags[=not_eq=urgent])
  gives ∃¬ a Core spelling; 'not' reserved in Appendix C
- B2: comparators are scalar predicates — contains is string containment,
  list handling comes only from §5.5 existential traversal
- B3: interpretation is schema-free; schema binding is a non-canonical
  execution concern; conformance vectors schema-free
- B4: semantic markers (type:, wildcard *, sort +/-) recognized on raw
  tokens; %2B sort recommendation removed
- S1: ABNF reconciled — standalone calls, chained-cond binding, seg-char/
  type-name defined, tokenization rules, trailing comma, chained lists
- S2: malformed literals / limit args / duplicate calls are syntax errors;
  round-trip numeral rule explicit
- S3: group-by removed from Core (reserved name, parse-rejected)
- S4/S5: nested projections always records mode; wildcard stems verbatim
- S6: determinism invariant scoped to desugaring, canonicalization deferred
  to §7; S7 covered by elem-cond
- §5.5: result multiplicity sentence (each record at most once)
- Appendix A: collection-matching and closed-converter rows added
- Appendix D: rows 10-12 (harper#2433 chained-leg coercion, #2434 index
  duplicates, contains numeric coercion); row 2/9 updated with PR #2437
  verification; examples de-personalized (ratings/reviews)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Negated-inner flattening exemption (§5.3/§6): elem-conds (ic.path=[]) and
  negated inner conditions are never normalized to plain Conditions.
  pushElementMatch + both bracket-singleton handlers now guard on
  !ic.negated && ic.path.length > 0.

- Elem-cond surface (§4 scoped-body): inside prop[...] (isScoped=true),
  =fiql-name=value with no preceding path gives Condition{path:[]};
  &=/|= after an elem-cond decomposes to conjunction + new elem-cond.

- Malformed typed literals are syntax errors (§5.2.2): boolean:yes,
  number:abc, number:, invalid date: → QueryError.

- Limit validation (§5.6): parseNonNegInt enforces non-negative decimal
  integer; 2-arg form requires end >= start.

- Duplicate call functions (§5.6): seenCalls Set detects second
  select()/sort()/limit() and throws QueryError.

- Nested tuple projection reserved (§5.7): select(rel{[x,y]}) → QueryError.

- Tests: rename ski-lengths example to ratings; add reviews[...] object-
  element test; pin chained-value-list, raw-token-marker, malformed-literal,
  limit-validation, and duplicate-call suites. 103/103 pass.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…nce ledger

not(body) is pure sugar desugared by De Morgan onto leaf negated flags and
ElementMatch.negated — canonical model unchanged, gives negated scopes a
surface (not(scores[=ge=10&=le=20])). Removed from Appendix C reserved
names; §5.6 carves it out of the call-function namespace; harper support
filed as HarperFast/harper#2441.

Appendix D is now a pointer: implementation divergence ledgers live with
the implementations (Harper: HarperFast/harper#2440), keeping the spec
vendor-independent. Also: singleton flattening clarified to include
element conditions (scores[=ge=10] ≡ scores=ge=10).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
not(...) (§5.4): term-form De Morgan negation, works at top level, inside
groups, and inside scoped bodies. negateGroup/negateTerm recurse inward:
Condition.negated toggles, and/or groups swap (De Morgan), ElementMatch.negated
toggles. Nested not() cancels. Empty not() is a QueryError. not is excluded
from the call-function namespace so unknown-call-function never fires on it.

Singleton ElementMatch flattening (§5.3/§5.5/§6): drop the path.length>0
guard from pushElementMatch and both bracket-singleton handlers. Plain
Conditions on list paths are already existential (§5.5), so elem-cond path=[]
flattens safely: scores[=ge=10] ≡ scores=ge=10. Negated inner and negated
scope remain unflattened. Update tests accordingly.

111/111 pass.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Maps PostgREST's URL filter syntax onto the canonical model: every filter
operator lands in Core or as an extension comparator, and both dialects'
logical layers are the same Group tree in prefix vs infix notation.
Records the real gaps (projection aliasing/casting, nulls ordering,
aggregates, resource embedding) and three semantic deltas to preserve
deliberately (embedded-filter join defaults, SQL null tri-state -> map RQL
negation to IS DISTINCT FROM, value-list any/all vs element scoping).

Serves as evidence the canonical representation is dialect-neutral and as
guidance for accepting a second surface syntax.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant