Differential conformance harness: Harper's REST parser vs. the RQL 2.0 reference parser

.gitignore

@@ -1,3 +1,5 @@
 downloaded-modules
 node_modules
 npm-debug.log
+conformance/*.actual
+conformance/*.tmp

conformance/README.md

@@ -0,0 +1,104 @@
+# Differential conformance harness
+
+Runs a corpus of RQL query strings through **both** Harper's REST query parser and the RQL 2.0
+reference parser in `src/`, maps Harper's output into the canonical model (§6) with an adapter,
+diffs the two, and classifies every difference.
+
+The output is [`conformance-report.md`](./conformance-report.md), which is committed. Spec
+Appendix D deliberately tracks no vendor rows — it links to each implementation's own public
+ledger — so this harness is what turns Harper's ledger
+([HarperFast/harper#2440](https://github.com/HarperFast/harper/issues/2440)) from a hand-written
+list into an exhaustive, re-runnable one.
+
+## Running it
+
+```bash
+npm run conformance                # replay the recorded fixture and regenerate the report
+npm run conformance:check          # fail if the committed report is out of date (CI)
+npm test                           # unit tests + an end-to-end replay assertion
+npm run typecheck:conformance      # type-checks conformance/, test/conformance/ and src/
+```
+
+None of those need a Harper checkout. Only re-recording does:
+
+```bash
+HARPER_PATH=../harper npm run conformance:record
+```
+
+`HARPER_PATH` must point at a Harper checkout that has been built (`npm install && npm run build`
+there) — the harness imports `dist/resources/search.js` and `dist/resources/RequestTarget.js`, and
+that import graph pulls in Table/rocksdb-js, so a full install is required. There is no default
+path and no implicit search: recording is an explicit act against a named checkout, and the
+checkout's commit is stamped into the fixture.
+
+Refresh the cached ledger before trusting a classification:
+
+```bash
+npm run conformance:refresh-ledger   # re-reads harper#2440 with the gh CLI
+```
+
+## Why record and replay
+
+Harper's parser keeps **module-global state** (`lastIndex` / `currentQuery` / `queryString` in
+`resources/search.ts`), so two parses must never interleave. Recording gives every query its own
+short-lived process, which makes that structural rather than a convention and contains a crash or
+a hang to the one case that caused it. Ordinary runs then replay the committed fixture, which
+means:
+
+- CI needs no cross-repo dependency and no build of Harper;
+- the report is reproducible — `conformance:check` asserts replay reproduces it byte for byte;
+- the adapter is exercised against real recorded host shapes rather than invented ones;
+- the Harper revision every classification was made against is recorded, not assumed.
+
+Replay uses **one** persistent reference-parser worker (the reference parser holds no global
+state), killed and replaced if a parse exceeds its wall-clock budget, so one pathological case
+costs that case and not the run.
+
+## What is in here
+
+| File | |
+|---|---|
+| `corpus.ts` | The declared corpus. Every case carries the grammar features it exercises and the ledger rows it witnesses. Ids are content hashes, so inserting a case never renumbers the others. |
+| `harperAdapter.ts` | Harper's parse output → the canonical model. Structural only: it resolves Harper's comparator vocabulary and shapes, and never re-interprets a value Harper already decoded. |
+| `tagged.ts` | Lossless, deterministic encoding of raw Harper output for the fixture — `URLSearchParams`, `Date`, `NaN`, `undefined`, arrays carrying marker properties. |
+| `canonical.ts` | Deterministic canonical JSON for a `ParseResult`, plus the structural differ the classification rules match on. |
+| `classify.ts` | The rules. Each maps the *shape* of a disagreement (or a named set of witness queries) to a ledger row, a proposed new row, or a reference-parser bug. |
+| `compare.ts` | Assembles a run from the fixture plus reference outcomes. Shared by the runner and the tests. |
+| `referenceRunner.ts` | Supervises the reference-parser worker: per-parse timeout, restart, and a settled outcome on every failure path. |
+| `report.ts` | Markdown rendering. Reads only committed data, never the wall clock, so the report is stable. |
+| `ledger.json` | A provenance-stamped **cache** of harper#2440. The issue stays canonical. |
+| `fixtures/harper-parse.json` | Recorded raw Harper output, stamped with the Harper commit, the reference commit, the Node version and the corpus digest. |
+| `../scripts/` | The CLI (`conformance-diff.mjs`), the two workers, and the ledger refresher. |
+
+## Adding a corpus case
+
+Add a `draft(...)` to the right group in `corpus.ts` with the grammar features it exercises, then:
+
+```bash
+HARPER_PATH=../harper npm run conformance:record
+```
+
+Replay refuses to run against a fixture recorded for a different corpus — the digest is checked —
+so there is no way to compare a new case against a stale recording.
+
+If the new case diverges for a reason no rule covers, the run **fails** and prints it. That is the
+design: a divergence nobody has classified is not allowed to sit quietly in the report.
+
+## Adding a classification rule
+
+Rules live in `classify.ts` and are tried in order. Prefer a rule that matches the *shape* of the
+disagreement (which pointers differ, and how) — it then covers every future case with the same
+root cause. Pin a rule to witness queries only where the shape is not distinctive.
+
+Each rule states a rationale citing the spec clause, and a `new` verdict must propose the ledger
+row to add. **The specification is never edited from here**; the report proposes rows and a human
+carries them to the ledger issue.
+
+Every rule must match at least one corpus case — `test/conformance/replay.test.ts` fails on a rule
+that matches nothing, so a rule made obsolete by a Harper fix has to be deleted along with it.
+
+## Scope
+
+This compares **parse** results. Value coercion against a table's declared attribute types,
+comparator evaluation and result materialization all happen later, in Harper's `resources/Table.ts`,
+and no case here can witness them — the report says so for the ledger rows that describe them.

conformance/canonical.ts

@@ -0,0 +1,163 @@
+/**
+ * Deterministic serialization of the canonical model, and a structural differ over it.
+ *
+ * Both parsers produce `ParseResult`s (Harper's via the adapter). Comparing them needs a
+ * form that is stable byte-for-byte across runs — object key order is not — and a diff
+ * that names *where* two results disagree, because the classification rules key off the
+ * shape of the disagreement rather than off the query string.
+ */
+
+import type { ParseResult, Value } from '../src/types.ts';
+
+export type Json = null | boolean | number | string | Json[] | { [key: string]: Json };
+
+/**
+ * Outcome of running one query string through one parser. A successful parse is carried as
+ * its canonical JSON view rather than as a `ParseResult`, so that an outcome crossing the
+ * worker IPC boundary is the same value the comparison and the report work on.
+ */
+export type Outcome =
+	| { status: 'parsed'; canonical: Json }
+	/** Rejected at parse — the canonical behavior for a syntax violation (§6.1). */
+	| { status: 'rejected'; error: string }
+	/** Rejected, but only into the pipeline: Harper's deferred mode (§6.1, ledger row 8). */
+	| { status: 'deferred-error'; error: string; canonical: Json }
+	/** The parse did not finish inside its wall-clock budget. */
+	| { status: 'timeout'; ms: number }
+	/** The parse produced a shape the adapter cannot place in the canonical model. */
+	| { status: 'adapter-gap'; error: string }
+	/** The parser process itself failed (crash, unusable output). */
+	| { status: 'harness-error'; error: string };
+
+/** Whether an outcome means "this query was not accepted", however that was signalled. */
+export function isRejection(outcome: Outcome): boolean {
+	return outcome.status === 'rejected' || outcome.status === 'deferred-error';
+}
+
+function canonicalValue(value: Value | undefined): Json {
+	if (value === undefined) return { $absent: true };
+	if (value === null) return null;
+	if (value instanceof Date)
+		return Number.isNaN(value.getTime()) ? { $date: 'invalid' } : { $date: value.toISOString() };
+	if (Array.isArray(value)) return value.map(canonicalValue);
+	if (typeof value === 'number' && !Number.isFinite(value)) return { $number: String(value) };
+	return value;
+}
+
+/**
+ * Canonical JSON view of a ParseResult. Absent optional members stay absent (they are not
+ * the same as present-and-empty), and values keep their type so `"3"` never reads as `3`.
+ */
+export function canonicalize(result: ParseResult): Json {
+	const out: { [key: string]: Json } = {};
+	if (result.filter !== undefined) out.filter = canonicalTerm(result.filter);
+	if (result.sort !== undefined) out.sort = result.sort.map((key) => ({ path: key.path, direction: key.direction }));
+	if (result.select !== undefined) out.select = canonicalProjection(result.select);
+	if (result.limit !== undefined) out.limit = canonicalValue(result.limit as Value);
+	if (result.offset !== undefined) out.offset = canonicalValue(result.offset as Value);
+	return out;
+}
+
+function canonicalProjection(projection: NonNullable<ParseResult['select']>): Json {
+	return {
+		mode: projection.mode,
+		fields: projection.fields.map((field) => {
+			const out: { [key: string]: Json } = { path: field.path };
+			if (field.projection) out.projection = canonicalProjection(field.projection);
+			return out;
+		}),
+	};
+}
+
+type Term = NonNullable<ParseResult['filter']>['terms'][number] | NonNullable<ParseResult['filter']>;
+
+function canonicalTerm(term: Term): Json {
+	if ('terms' in term) return { kind: 'group', operator: term.operator, terms: term.terms.map(canonicalTerm) };
+	if ('some' in term) {
+		const out: { [key: string]: Json } = { kind: 'elementMatch', path: term.path };
+		if (term.negated) out.negated = true;
+		out.some = canonicalTerm(term.some);
+		return out;
+	}
+	const out: { [key: string]: Json } = { kind: 'condition', path: term.path, comparator: term.comparator };
+	if (term.negated) out.negated = true;
+	out.value = canonicalValue(term.value);
+	return out;
+}
+
+/** Keys are emitted sorted, so two equal structures serialize to identical bytes. */
+export function stableStringify(value: Json, indent = 0): string {
+	const render = (node: Json, depth: number): string => {
+		const pad = indent ? '\n' + ' '.repeat(indent * (depth + 1)) : '';
+		const closePad = indent ? '\n' + ' '.repeat(indent * depth) : '';
+		if (node === null || typeof node !== 'object') return JSON.stringify(node);
+		if (Array.isArray(node)) {
+			if (node.length === 0) return '[]';
+			return '[' + node.map((item) => pad + render(item, depth + 1)).join(',') + closePad + ']';
+		}
+		const keys = Object.keys(node).sort();
+		if (keys.length === 0) return '{}';
+		return '{' + keys.map((key) => pad + JSON.stringify(key) + ':' + (indent ? ' ' : '') + render(node[key], depth + 1)).join(',') + closePad + '}';
+	};
+	return render(value, 0);
+}
+
+/**
+ * Short content fingerprint of a canonical value. The report prints truncated JSON for
+ * readability; the fingerprint is what makes the committed bytes change when a value changes
+ * past the truncation point, so `--check` cannot miss a regression it did not have room to show.
+ */
+export function digest(value: Json): string {
+	const text = stableStringify(value);
+	let hash = 0x811c9dc5;
+	for (let index = 0; index < text.length; index++) {
+		hash ^= text.charCodeAt(index);
+		hash = Math.imul(hash, 0x01000193) >>> 0;
+	}
+	return hash.toString(16).padStart(8, '0');
+}
+
+export type Difference = {
+	/** JSON-pointer-ish location, e.g. `/filter/terms/0/value`. */
+	at: string;
+	kind: 'value' | 'type' | 'ref-only' | 'harper-only';
+	ref?: Json;
+	harper?: Json;
+};
+
+const typeOf = (node: Json): string => (node === null ? 'null' : Array.isArray(node) ? 'array' : typeof node);
+
+export function diffCanonical(ref: Json, harper: Json): Difference[] {
+	const differences: Difference[] = [];
+	walk('', ref, harper, differences);
+	return differences;
+}
+
+function walk(at: string, ref: Json, harper: Json, out: Difference[]): void {
+	if (typeOf(ref) !== typeOf(harper)) {
+		out.push({ at, kind: 'type', ref, harper });
+		return;
+	}
+	if (ref === null || typeof ref !== 'object') {
+		if (ref !== harper) out.push({ at, kind: 'value', ref, harper });
+		return;
+	}
+	if (Array.isArray(ref) && Array.isArray(harper)) {
+		const length = Math.max(ref.length, harper.length);
+		for (let index = 0; index < length; index++) {
+			const location = `${at}/${index}`;
+			if (index >= harper.length) out.push({ at: location, kind: 'ref-only', ref: ref[index] });
+			else if (index >= ref.length) out.push({ at: location, kind: 'harper-only', harper: harper[index] });
+			else walk(location, ref[index], harper[index], out);
+		}
+		return;
+	}
+	const refObject = ref as { [key: string]: Json };
+	const harperObject = harper as { [key: string]: Json };
+	for (const key of [...new Set([...Object.keys(refObject), ...Object.keys(harperObject)])].sort()) {
+		const location = `${at}/${key}`;
+		if (!(key in harperObject)) out.push({ at: location, kind: 'ref-only', ref: refObject[key] });
+		else if (!(key in refObject)) out.push({ at: location, kind: 'harper-only', harper: harperObject[key] });
+		else walk(location, refObject[key], harperObject[key], out);
+	}
+}