From f6165299b17d07818439649081165e7228b2d417 Mon Sep 17 00:00:00 2001 From: JonnyTran Date: Mon, 20 Jul 2026 18:23:51 -0700 Subject: [PATCH 01/21] =?UTF-8?q?docs(spec):=20extraction=20table=20design?= =?UTF-8?q?=20=E2=80=94=20workspace=20projection,=20Perspective=20grid,=20?= =?UTF-8?q?review-page=20retirement?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../2026-07-20-extraction-table-design.md | 195 ++++++++++++++++++ 1 file changed, 195 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-20-extraction-table-design.md diff --git a/docs/superpowers/specs/2026-07-20-extraction-table-design.md b/docs/superpowers/specs/2026-07-20-extraction-table-design.md new file mode 100644 index 000000000..22167bcc3 --- /dev/null +++ b/docs/superpowers/specs/2026-07-20-extraction-table-design.md @@ -0,0 +1,195 @@ +# Extraction Table — Design Spec + +**Date:** 2026-07-20 +**Branch:** `feat/v2-ui-extraction-table` (based on `develop` @ `52eab556f`, PR #232 merged) +**Predecessors:** `2026-06-27-schema-centric-data-model-design.md` (§17.4 projection), +`2026-07-09-v2-frontend-vertical-slice-design.md` (§7 review form + ledger), +`2026-07-19-reference-review.md` (interrogation brief that triggered this redesign). + +## 1. Problem + +The v2 review slice (#230/#232) shipped a per-reference review page whose client-side +assembly (`entities/review/ReferenceReview` + `get-reference-review-use-case`) composes +projection + questions + records + versions + suggestions + responses across 5+ +endpoints per reference. That is exactly the chattiness §7's ledger item predicted +("if the multi-endpoint composition proves chatty, enrich the projection payload +server-side"), and it duplicates derivation that belongs on the server: review state +**is** submitted suggestions/responses — it should not exist as a parallel +client-side entity. + +Meanwhile the product's first-order surface — the view a user actually starts from — +is missing: a workspace-level **extraction table** aggregating references (rows) × +schemas (column groups), with cell values coalesced over suggestions/responses and +cells linking to their source record. + +This build executes the ledger item (server-side enrichment), ships the extraction +table, and deletes the reference-review page whose function is superseded (review +embeds into the annotation + projection flow going forward). + +## 2. Inherited decisions (from prior specs — restated, not reopened) + +- **§17.4:** the projection view is *the* product surface; precedence + `submitted response → suggestion → empty` is applied **server-side**; the endpoint + contract reserves a backing swap (OLAP/materialized) without shape changes. +- **Contract gotchas** that move from client to server-side derivation: asymmetric + keying (responses by question *name*, suggestions by question *id* — join through + the questions list), double-wrapped `{question_name: {value}}` response values, + `GET /records/{id}/responses` returning bare object **or literal `null`** with 200. +- **Reuse-don't-fork:** widget adapters, `ReviewCellInput`, `ReviewProvenance`, and + the submit/draft/discard use-cases are the reuse base for the future embedded + review drawer. v0's Tabulator `RenderTable` lineage is **not** reused — Perspective + replaces it. +- **DDD layering** as in the rest of `v2/`: repository → use-case → Pinia storage → + page/composable; openapi-typescript types via the `gen:api` drift gate. + +## 3. Decisions (from 2026-07-20 interview) + +### 3.1 Grid semantics + +| Decision | Choice | +|---|---| +| Rows | Every distinct reference in the workspace (union across schemas' records); schemas with no record for a reference render empty cells — the grid doubles as a coverage map | +| Columns | Per-schema column groups; **all** question columns shown (visibility config deferred); flat names `Schema.question` / `Schema.question.subcol` | +| Cell content | Coalesced data values: latest **submitted** response (any user) `??` suggestion `??` empty. Drafts never appear — endpoint stays user-agnostic and cacheable | +| Record multiplicity | One **effective record** per (reference, schema); row multiplicity comes only from `table`-type question values fanning out into N sub-rows | +| Multi-table alignment | Independent stacking: row count = max(fan-outs) per reference; no cartesian product, no fabricated row pairing across unrelated table questions | +| Fan-out scalars | **Repeat scalar values on every fan-out row** (true denormalized rows — correct for pivots/Arrow/CSV later); visual grouping via row-banding by reference, not rowspan (Perspective is flat-columnar and cannot merge cells) | +| Scale target | ~100s of references (typical review), ~5 schemas, ~30 columns; one paginated endpoint, no virtualization work needed (Perspective handles far more) | +| Sorting/filtering | None — static grid in v1 | + +### 3.2 Backend — workspace-level denormalized projection + +**New endpoint** `GET /api/v2/projection?workspace_id=…&offset=0&limit=50` +(reference-page granularity; `limit` counts references, not fan-out rows): + +```jsonc +{ + "columns": [ + { "name": "Design.type", "schema_id": "…", "schema_name": "Design", + "question_name": "type", "sub_column": null, "dtype": "…" }, + { "name": "Outcomes.results.value", "schema_id": "…", "schema_name": "Outcomes", + "question_name": "results", "sub_column": "value", "dtype": "…" } + ], + "rows": [ + { "reference": "10.1234/abc", "row_index": 0, + "cells": { "Design.type": { "value": "RCT", "source": "response", + "record_id": "…" }, + "Outcomes.results.value": { "value": "12%", "source": "suggestion", + "record_id": "…", "agent": "gpt-x", + "score": 0.92 } } } + ], + "total_references": 213 +} +``` + +- Server performs the full denormalization (coalesce → table fan-out → independent + stacking → scalar repetition); the client only renders. +- **Enriched cells** carry `record_id` + `source` (+ `agent`/`score` when the value + came from a suggestion) so any consumer links and shows provenance with zero + extra calls. +- Absent cells (no record / no value) are omitted from `cells`; the client renders + them as null. +- **Assembly must be batch-queried** — all schemas → all records in workspace → + bulk latest-submitted-responses + suggestions — not the per-record N+1 the + 2026-07-19 brief flagged in `contexts/v2/projection.py`. +- The per-reference `GET /projection/references/{reference:path}` **stays** (SDK #231 + consumes it) and adopts the same enriched cell shape **additively** + (`record_id`/`agent`/`score` added to `ProjectionCell`). +- The `/projection/…` prefix convention is kept (avoids the greedy + `GET /references/{reference:path}` shadowing). + +### 3.3 Frontend — `/extractions` page on Perspective + +- **New route `pages/extractions/index.vue`** ("Extractions") — full-page standalone + view; **not** wired into `index.vue`/nav in this build. (`/schemas/*` remains + reserved for schema-editor functions.) +- **Grid engine:** Perspective **4.x** under the `@perspective-dev/*` npm scope + (`@perspective-dev/client`, `@perspective-dev/viewer`, + `@perspective-dev/viewer-datagrid`) — the actively-developed line matching + . + `@finos/*` is frozen at 3.8.0 and not used. + - ESM WASM bootstrap via Vite `?url` imports: + `perspective.init_server(fetch(SERVER_WASM))` + + `perspective_viewer.init_client(fetch(CLIENT_WASM))`; Vite `target: esnext`. + - Lazy-loaded route chunk so the WASM/bundle cost is paid only on this page. +- **Data path:** client pages through the endpoint (50–100 refs/page) and loads all + pages into **one** Perspective table as flat JSON records keyed by the + `columns[].name` manifest. Arrow IPC streaming is a recorded follow-up (§5), not + built now. +- **Static grid:** viewer toolbar/settings hidden; no sort/filter UI; row-banding by + reference. +- **Interaction v1:** native row hover + pointer cursor on linkable cells. + Cell click emits the **future annotation URL contract** + `/dataset/{schema_id}/annotation-mode?_search={reference}` (schema id in the + dataset slot) behind a feature guard — **disabled** until annotation-mode resolves + v2 schema ids (ledger, §5). Click plumbing (`record_id`, `schema_id`, `reference` + per cell) ships and is testable now. +- **Layering:** `ProjectionRepository.getWorkspaceProjection()` → + `get-workspace-projection-use-case` → Pinia storage → `useExtractionsViewModel` → + page. + +### 3.4 Deletions — review derives from suggestions/responses, no parallel entity + +The reference-review **page is deleted**; its function is superseded by the +extraction table now and the embedded annotation/review flow later. + +**Delete in this build:** +- `pages/references/[...reference].vue` + `useReferenceReviewViewModel` (+ test) +- `components/v2/review/ProjectionReviewForm.vue`, `ReviewRecordCard.vue` +- `v2/domain/entities/review/ReferenceReview.ts` +- `v2/domain/usecases/get-reference-review-use-case.ts` (+ test) — superseded by the + server-side enriched projection +- `v2/infrastructure/storage/ReferenceReviewsStorage.ts` (page-scoped cache, dies + with the page) +- The `/references/…` entry link in `components/v2/schemas/V2RecordsTable.vue` +- e2e specs `e2e/v2/{review-loop,draft-lifecycle,slashed-reference}.spec.ts` +- Corresponding `v2/di/di.ts` registrations + +**Keep (powers the future embedded-review drawer):** +- `entities/review/widget-adapters.ts`, `widget-mapping.ts`, `SuggestionHint.ts`, + `response-values.ts` (each with tests) +- `components/v2/review/ReviewCellInput.vue`, `ReviewProvenance.vue` +- `submit-reference-review-use-case.ts`, `save-review-draft-use-case.ts`, + `discard-review-use-case.ts` + +**Accepted risk:** deleting the three e2e specs removes end-to-end coverage of the +suggestion→response submit path until the drawer lands (kept use-cases retain unit +tests). Replacement gate in this build: a new `e2e/v2/extractions-grid.spec.ts` +(seed → grid renders coalesced values → empty cells correct). + +## 4. Testing strategy + +- **Server unit/integration:** denormalization (coalesce precedence, table fan-out, + independent stacking, scalar repetition, empty-cell omission), row universe + (union of references, refs with zero records in a schema), pagination + (`total_references`, offset/limit on references), enriched-cell provenance, + latest-submitted-wins across users, batch query count (guard against N+1). +- **Frontend unit (vitest):** repository/use-case against the new contract; grid + row/column adapter (manifest → Perspective schema + rows); click-URL contract + builder; guard behavior. +- **Contract:** `gen:api` drift gate regenerated for the new endpoint + enriched + `ProjectionCell`. +- **e2e (Playwright):** `extractions-grid.spec.ts` as above; existing v2 e2e suite + minus the three deleted specs stays green. + +## 5. Ledger (recorded, not built) + +- **Arrow IPC streaming** from server to Perspective (the reason Perspective was + chosen; contract already columnar-friendly). +- **Annotation-mode v2 upgrade** — resolve v2 schema ids at + `/dataset/{schema_id}/annotation-mode`; flips the cell-click guard on. +- **Embedded review drawer** on the extraction table (reuses kept `ReviewCellInput`, + widget adapters, submit/draft/discard use-cases); restores e2e submit-path + coverage. +- **Column visibility config** (which columns per schema appear), then sort/filter. +- **Record-extent hover glow + provenance tooltip** via the datagrid style listener. +- **Migrate `V2RecordsTable`/search surfaces** onto enriched projection cells where + useful. +- Nav/`index.vue` integration of `/extractions`. + +## 6. Scope boundary + +In: new workspace projection endpoint + enriched `ProjectionCell`, `/extractions` +Perspective page, deletions per §3.4, tests per §4. +Out: everything in §5; any v1 dataset/annotation-mode changes; SDK changes beyond +regenerated types (additive cell fields are backward-compatible for #231). From 8179d65f96a75266cbd2adbe98ff3c6106e22057 Mon Sep 17 00:00:00 2001 From: JonnyTran Date: Mon, 20 Jul 2026 18:32:00 -0700 Subject: [PATCH 02/21] docs(spec): add Perspective/Vue3/Nuxt4 integration refs + single-row table-value constraint --- .../2026-07-20-extraction-table-design.md | 81 ++++++++++++++++++- 1 file changed, 78 insertions(+), 3 deletions(-) diff --git a/docs/superpowers/specs/2026-07-20-extraction-table-design.md b/docs/superpowers/specs/2026-07-20-extraction-table-design.md index 22167bcc3..84f8e3850 100644 --- a/docs/superpowers/specs/2026-07-20-extraction-table-design.md +++ b/docs/superpowers/specs/2026-07-20-extraction-table-design.md @@ -127,8 +127,42 @@ embeds into the annotation + projection flow going forward). - **Layering:** `ProjectionRepository.getWorkspaceProjection()` → `get-workspace-projection-use-case` → Pinia storage → `useExtractionsViewModel` → page. +- **Vue 3 / Nuxt 4 integration notes** (verified against this repo's config): + - The app is `ssr: false` (SPA) — no `` wrapper needed; the WASM + bootstrap just runs once client-side (page/component `onMounted` or a lazy + module-level init guard). + - `` is a **web Custom Element**: Vue must be told not to + resolve it as a component via + `vue.compilerOptions.isCustomElement = (tag) => tag.startsWith("perspective-")` + in `nuxt.config.ts` (see §7 refs). + - Vite needs `build.target`/`optimizeDeps.esbuildOptions.target: "esnext"` for + the Perspective ESM/WASM builds, and the `?url` asset-import syntax for the + two `.wasm` binaries (§7 refs). + - There is **no official Vue/Nuxt Perspective example** — the repo's + `vite-example` (bundler wiring) and `react-example` (wrapping the custom + element in a component framework) are the transferable templates (§7). -### 3.4 Deletions — review derives from suggestions/responses, no parallel entity +### 3.4 Discovered constraint — table-question values are single-row today + +Found during spec refinement (2026-07-20), affects the fan-out design: + +- The server validator (`validators/v2/values.py::_validate_table`) accepts a table + value only as **a dict keyed by bound columns** — i.e. **one row**. +- The frontend editor agrees: `V2TableEditor.vue` renders + `data: [{ ...modelValue }]` — "single-row grid: the value IS one dict". +- A table question's **sub-columns are its `V2Question.columns` binding** (the ≥1 + schema columns it binds; non-table types bind exactly 1). Denormalized column + names `Schema.question.subcol` come from this list. + +**Resolution (recommended, in scope for this build's backend task):** extend the +table value contract **additively** to also accept `list[dict]` (N rows), keeping +bare `dict` valid as the 1-row case — `_validate_table` validates each row's keys +against the binding; `V2TableEditor` keeps emitting the dict form until the +embedded-review work touches it. The projection fan-out handles both shapes from +day one (`dict → [dict]` normalization), so the grid is correct now (fan-out = 1) +and automatically right when multi-row values start arriving from the SDK/agents. + +### 3.5 Deletions — review derives from suggestions/responses, no parallel entity The reference-review **page is deleted**; its function is superseded by the extraction table now and the embedded annotation/review flow later. @@ -189,7 +223,48 @@ tests). Replacement gate in this build: a new `e2e/v2/extractions-grid.spec.ts` ## 6. Scope boundary -In: new workspace projection endpoint + enriched `ProjectionCell`, `/extractions` -Perspective page, deletions per §3.4, tests per §4. +In: new workspace projection endpoint + enriched `ProjectionCell`, table-value +`list[dict]` extension (§3.4), `/extractions` Perspective page, deletions per §3.5, +tests per §4. Out: everything in §5; any v1 dataset/annotation-mode changes; SDK changes beyond regenerated types (additive cell fields are backward-compatible for #231). + +## 7. Integration references — Perspective × Vue 3 / Nuxt 4 + +Perspective 4.x (`@perspective-dev/*`, all at 4.5.2 as of 2026-07-20): + +- **Importing & WASM bootstrap** (the page this design's bundling follows): + — + Vite `?url` imports of `perspective-server.wasm` + `perspective-viewer.wasm`, + `perspective.init_server(fetch(…))` / `perspective_viewer.init_client(fetch(…))`, + Vite `target: esnext`. +- **Loading data**: — + `client.table(data | schema)`, `viewer.load(table)`, sharing one `Table` across + viewers (relevant if the review drawer later mounts a second viewer). +- **Guide root / architecture**: +- **`@perspective-dev/viewer` web-component API** (attributes, `restore()` for + locking the static config, plugin selection, themes): + → "JavaScript → `@perspective-dev/viewer` + Web Component". +- **Bundler example (closest to our Nuxt 4/Vite setup)**: + +- **Framework-wrapper example (pattern for our Vue wrapper component)**: + + — note there is **no official Vue/Nuxt example**; the React one shows the + custom-element lifecycle (load on mount, `delete()` on unmount) to mirror in + `onMounted`/`onBeforeUnmount`. + +Vue 3 / Nuxt 4 custom-element integration: + +- **Vue 3 — using web components in Vue**: + — `compilerOptions.isCustomElement` + so `` isn't resolved as a Vue component; passing DOM + properties vs attributes. +- **Nuxt 4 — vue compiler options in `nuxt.config.ts`**: + (`vue.compilerOptions`). +- **Nuxt 4 — client components** (background; not needed here since `ssr: false`): + +- **Vite — explicit `?url` asset imports** (WASM binaries): + +- **Vite — `build.target`** (`esnext` requirement): + From 7675fbb7b139d18e4756793bc143d47220e37ef1 Mon Sep 17 00:00:00 2001 From: JonnyTran Date: Mon, 20 Jul 2026 22:51:06 -0700 Subject: [PATCH 03/21] plan --- .../plans/2026-07-20-extraction-table.md | 2316 +++++++++++++++++ 1 file changed, 2316 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-20-extraction-table.md diff --git a/docs/superpowers/plans/2026-07-20-extraction-table.md b/docs/superpowers/plans/2026-07-20-extraction-table.md new file mode 100644 index 000000000..fb86d3c15 --- /dev/null +++ b/docs/superpowers/plans/2026-07-20-extraction-table.md @@ -0,0 +1,2316 @@ +# Extraction Table Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Ship a workspace-level denormalized extraction table — new `GET /api/v2/projection` endpoint with enriched provenance cells, a `/extractions` page rendered by Perspective 4.x, an additive `list[dict]` table-value contract, and deletion of the superseded reference-review page. + +**Delivery: two phases, two PRs.** +- **Phase 1 (Tasks 1–7, PR 1):** everything additive and low-risk — server contract (validator extension, enriched cells, workspace endpoint) plus the frontend data layer (gen:api, domain types, repository, grid adapter, use-case, storage, DI). No UI, no deletions, no new dependencies. Mergeable to `develop` on its own; SDK #231 and any other consumer can build against the endpoint immediately. +- **Phase 2 (Tasks 8–13, PR 2, stacked on Phase 1):** the integration-risk half — Perspective deps/WASM/Vite wiring, the custom-element grid wrapper, the `/extractions` page, deletion of the reference-review page, and the e2e gate. If Perspective 4.5.2 fights back (WASM boot, init API drift, custom-element quirks), Phase 1 is already merged and unaffected. + +**Architecture:** The server performs the full denormalization (coalesce `submitted response → suggestion → empty`, table fan-out, independent stacking, scalar repetition) in `contexts/v2/projection.py` using batched queries; the client only pages, aggregates, and renders. Frontend follows the existing v2 DDD chain: `ProjectionRepository` → `GetWorkspaceProjectionUseCase` → Pinia storage → `useExtractionsViewModel` → `pages/extractions/index.vue`, with a Perspective web-component grid wrapped in `ExtractionsGrid.vue`. + +**Tech Stack:** FastAPI + SQLAlchemy async (extralit-server), Vue 3 / Nuxt 4 / Pinia / ts-injecty (extralit-frontend), Perspective `@perspective-dev/*` 4.5.2 (WASM), openapi-typescript contract gate, pytest / vitest / Playwright. + +**Spec:** `docs/superpowers/specs/2026-07-20-extraction-table-design.md` (this plan implements §3–§4 within the §6 scope boundary; §5 items are explicitly NOT built). + +## Global Constraints + +- Branching: Phase 1 lands on `feat/v2-ui-extraction-table` (based on `develop` @ `52eab556f`) and PRs to `develop`. Phase 2 starts on a stacked branch `feat/v2-ui-extraction-grid` created from the Phase 1 branch (rebase onto `develop` once PR 1 merges; PR 2 targets `develop`). Commit per task, conventional-commit style (`feat(server): …`, `feat(v2-ui): …`, `test(…): …`). +- Python: **uv only** (`uv run pytest`, `uv add`); never pip/poetry. All server commands run from `extralit-server/`. All frontend commands run from `extralit-frontend/`. +- Perspective packages: `@perspective-dev/client`, `@perspective-dev/server`, `@perspective-dev/viewer`, `@perspective-dev/viewer-datagrid` — all pinned **4.5.2**. `@finos/*` is forbidden (frozen at 3.8.0). +- **Reuse-don't-fork** (spec §2): reuse `ProjectionRepository`, `useStoreFor` (v1 store factory), `useResolve`/ts-injecty DI, `InternalPage`/`AppHeader` layout, `e2e/v2/fixtures.ts`. v0's Tabulator `RenderTable` lineage is NOT reused. Kept for the future review drawer (do not delete): `widget-adapters.ts`, `widget-mapping.ts`, `SuggestionHint.ts`, `response-values.ts`, `ReviewCellInput.vue`, `ReviewProvenance.vue`, `submit-reference-review-use-case.ts`, `save-review-draft-use-case.ts`, `discard-review-use-case.ts` (each with tests). +- **gen:api drift gate:** any server contract change requires `npm run gen:api` from `extralit-frontend/` and committing BOTH `v2/infrastructure/api/openapi.json` and `v2/infrastructure/api/generated/v2-api.ts`. Repositories must type against `components["schemas"][…]` from the generated file. +- Frontend TS: `isolatedModules` — type-only imports MUST use the inline form `import { type X } from "…"`. +- Pinia store key = the state class's constructor name (`useStoreFor(Ctor)` → `defineStore(Ctor.name, …)`); the class name must be unique across ALL v1+v2 stores. +- Server integration tests need live services. On the Orin host: Postgres :5432, MinIO :9000, Elasticsearch :9200 already publish to localhost; Redis does NOT — start a throwaway one first: `docker run -d --rm --name tmp-redis -p 6379:6379 redis:7` (the autouse `empty_job_queues` fixture needs it even for non-queue tests). +- Frontend unit tests: `npx vitest run ` for a single file, `npm run test` for the suite. Typecheck: `npx nuxi typecheck`. Lint: `npm run lint`. + +## File Structure + +**extralit-server (modify):** +- `src/extralit_server/validators/v2/values.py` — `_validate_table` accepts `list[dict]` additively (Task 1) +- `src/extralit_server/api/schemas/v2/projection.py` — enrich `ProjectionCell`; add `WorkspaceProjection*` models (Tasks 2, 3) +- `src/extralit_server/contexts/v2/projection.py` — enrich `build_reference_view`; add `build_workspace_view` + pure denormalization helpers (Tasks 2, 3) +- `src/extralit_server/api/v2/projection.py` — add `GET /projection` route (Task 4) +- `tests/unit/validators/v2/test_values.py`, `tests/integration/contexts/v2/test_projection.py`, `tests/integration/api/v2/test_projection.py` — extend; new `tests/integration/contexts/v2/test_workspace_projection.py` + +**extralit-frontend (create):** +- `v2/domain/entities/projection/WorkspaceProjection.ts` — domain types (columns/cells/rows/aggregate) +- `v2/domain/entities/projection/grid-adapter.ts` (+`.test.ts`) — pure: manifest→flat rows, band parity, cell lookup, annotation-URL contract + guard +- `v2/domain/usecases/get-workspace-projection-use-case.ts` (+`.test.ts`) — pages through endpoint, saves to storage +- `v2/infrastructure/storage/ExtractionsStorage.ts` — `useExtractions` Pinia store +- `components/v2/extractions/perspective-bootstrap.ts` — WASM init singleton (`__mocks__/perspective-bootstrap.js` stub) +- `components/v2/extractions/ExtractionsGrid.vue` (+`.test.ts`) — Perspective viewer wrapper: load, banding, click plumbing +- `pages/extractions/index.vue`, `pages/extractions/useExtractionsViewModel.ts` (+`.test.ts`) — the `/extractions` route +- `e2e/v2/extractions-grid.spec.ts` — replacement e2e gate + +**extralit-frontend (modify):** `v2/infrastructure/repositories/ProjectionRepository.ts` (+test), `v2/di/di.ts`, `nuxt.config.ts`, `vitest.config.ts`, `package.json`, `translation/{en,de,es,ja}.js`, `composables/useV2Breadcrumbs.ts`, `components/v2/schemas/V2RecordsTable.vue` (+test), `e2e/v2/fixtures.ts`, `e2e/v2/seed/seed_v2_e2e.py`, new `v2/domain/entities/review/ReviewCell.ts` + import updates in kept review files. + +**extralit-frontend (delete, Task 11):** `pages/references/[...reference].vue`, `pages/references/useReferenceReviewViewModel.ts` (+`.test.ts`), `components/v2/review/ProjectionReviewForm.vue` (+`.test.ts` if present), `components/v2/review/ReviewRecordCard.vue` (+test if present), `v2/domain/entities/review/ReferenceReview.ts`, `v2/domain/usecases/get-reference-review-use-case.ts` (+`.test.ts`), `v2/infrastructure/storage/ReferenceReviewsStorage.ts`, `e2e/v2/{review-loop,draft-lifecycle,slashed-reference}.spec.ts`. + +--- + +## Phase 1 — Contract + data layer (PR 1: additive, no UI, no deletions) + +Tasks 1–7. Server-side table-value extension, enriched provenance, the workspace projection endpoint, and the frontend chain up through DI registration (`ProjectionRepository.getWorkspaceProjection` → grid adapter → `GetWorkspaceProjectionUseCase` → `ExtractionsStorage`). Nothing here touches Perspective, pages, or existing UI — every change is additive and independently shippable. + +### Task 1: Table-value `list[dict]` contract extension (server) + +**Files:** +- Modify: `extralit-server/src/extralit_server/validators/v2/values.py` (`_validate_table`, lines ~48–57) +- Test: `extralit-server/tests/unit/validators/v2/test_values.py` + +**Interfaces:** +- Consumes: existing `V2ResponseValueValidator.validate(value, *, type, settings, columns)` dispatch (unchanged). +- Produces: `_validate_table` accepting `dict` (1-row case, unchanged) OR `list[dict]` (N rows); every row's keys validated against the `columns` binding. Task 3's fan-out relies on both shapes being storable. + +- [ ] **Step 1: Write the failing tests** + +Append to `tests/unit/validators/v2/test_values.py` (it already has `TABLE_SETTINGS = {"type": "table"}` and imports `V2ResponseValueValidator`, `QuestionType`, `UnprocessableEntityError`, `pytest`): + +```python +def test_table_value_accepts_list_of_row_dicts(): + V2ResponseValueValidator.validate( + [{"a": 1}, {"a": 2, "b": "x"}], type=QuestionType.table, settings=TABLE_SETTINGS, columns=["a", "b"] + ) + + +def test_table_value_accepts_empty_list(): + V2ResponseValueValidator.validate([], type=QuestionType.table, settings=TABLE_SETTINGS, columns=["a"]) + + +def test_table_value_list_rejects_unbound_keys_in_any_row(): + with pytest.raises(UnprocessableEntityError, match="not bound"): + V2ResponseValueValidator.validate( + [{"a": 1}, {"z": 2}], type=QuestionType.table, settings=TABLE_SETTINGS, columns=["a", "b"] + ) + + +def test_table_value_list_rejects_non_dict_rows(): + with pytest.raises(UnprocessableEntityError, match="dict of values per row"): + V2ResponseValueValidator.validate( + [{"a": 1}, 5], type=QuestionType.table, settings=TABLE_SETTINGS, columns=["a"] + ) +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `cd extralit-server && uv run pytest tests/unit/validators/v2/test_values.py -v` +Expected: the 4 new tests FAIL (`list` currently raises "table question expects a dict"); existing tests PASS. + +- [ ] **Step 3: Implement** + +Replace `_validate_table` in `src/extralit_server/validators/v2/values.py`: + +```python + @staticmethod + def _validate_table(value, columns: list[str]) -> None: + # Additive contract (spec §3.4): a bare dict is the 1-row case; list[dict] is N rows. + rows = value if isinstance(value, list) else [value] + bound = set(columns) + for row in rows: + if not isinstance(row, dict): + raise UnprocessableEntityError( + f"table question expects a dict of values per row, found {type(row)}" + ) + extra = sorted(k for k in row if k not in bound) + if extra: + raise UnprocessableEntityError( + f"table value keys {extra!r} are not bound columns; bound: {sorted(bound)!r}" + ) +``` + +- [ ] **Step 4: Run the full validator test file** + +Run: `cd extralit-server && uv run pytest tests/unit/validators/v2/test_values.py -v` +Expected: ALL PASS (the pre-existing `test_table_value_keys_must_be_subset_of_columns` matches `"not bound"` and still passes; the pre-existing non-dict error message changed from "expects a dict of values, found" to "…per row, found" — if any existing test matched the old wording, update its `match=` to `"dict of values per row"`). + +- [ ] **Step 5: Commit** + +```bash +git add extralit-server/src/extralit_server/validators/v2/values.py extralit-server/tests/unit/validators/v2/test_values.py +git commit -m "feat(server): accept list[dict] table values additively (spec §3.4)" +``` + +--- + +### Task 2: Enrich per-reference `ProjectionCell` with provenance (server) + +**Files:** +- Modify: `extralit-server/src/extralit_server/api/schemas/v2/projection.py` +- Modify: `extralit-server/src/extralit_server/contexts/v2/projection.py` (`build_reference_view`) +- Test: `extralit-server/tests/integration/contexts/v2/test_projection.py`, `extralit-server/tests/integration/api/v2/test_projection.py` + +**Interfaces:** +- Produces: `ProjectionCell` gains optional `record_id: UUID | None`, `agent: str | None`, `score: float | list[float] | None` — **additive**, so SDK #231 stays backward-compatible. Populated: response cells → `record_id`; suggestion cells → `record_id` + `agent` + `score`; empty cells → all `None`. + +- [ ] **Step 1: Write the failing tests** + +In `tests/integration/contexts/v2/test_projection.py`, extend the existing suggestion-cell test's assertions (find the test asserting `source == "suggestion"`) and add assertions to the response-precedence test. Add to the suggestion test (the factory default agent/score come from the explicit kwargs you pass — pass them): + +```python + # in the suggestion-cell test, create the suggestion with explicit provenance: + await V2SuggestionFactory.create(record=record, question=q, value="flu", agent="gpt-x", score=0.92) + ... + assert cell.record_id == record.id + assert cell.agent == "gpt-x" + assert cell.score == 0.92 +``` + +In the response-over-suggestion test add: + +```python + assert cell.record_id == record.id + assert cell.agent is None and cell.score is None +``` + +In `tests/integration/api/v2/test_projection.py`, extend `test_projection_view_resolves_suggestion_cell`: pass `agent="e2e-agent", score=0.5` to `V2SuggestionFactory.create(...)` and assert on the JSON cell: + +```python + assert cell["record_id"] == str(record.id) + assert cell["agent"] == "e2e-agent" + assert cell["score"] == 0.5 +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cd extralit-server && uv run pytest tests/integration/contexts/v2/test_projection.py tests/integration/api/v2/test_projection.py -v` +Expected: FAIL — `ProjectionCell` has no attribute `record_id`. (Needs services up; see Global Constraints for the throwaway redis.) + +- [ ] **Step 3: Implement** + +`src/extralit_server/api/schemas/v2/projection.py` — extend `ProjectionCell`: + +```python +class ProjectionCell(BaseModel): + question_name: str + value: Any | None = None + source: Literal["response", "suggestion"] | None = None # None => neither exists yet + # Enriched provenance (spec §3.2): consumers link and attribute with zero extra calls. + record_id: UUID | None = None + agent: str | None = None + score: float | list[float] | None = None +``` + +`src/extralit_server/contexts/v2/projection.py` — in `build_reference_view`, keep whole suggestion rows instead of just values, and populate the new fields. Change the suggestions dict comprehension: + +```python + suggestions = {(s.record_id, s.question_id): s for s in sugg_rows} +``` + +and the cell-building branches: + +```python + wrapped = responses.get(record.id, {}).get(question.name) + if wrapped is not None: + cells.append( + ProjectionCell( + question_name=question.name, + value=wrapped.get("value"), + source="response", + record_id=record.id, + ) + ) + elif (record.id, question.id) in suggestions: + suggestion = suggestions[(record.id, question.id)] + cells.append( + ProjectionCell( + question_name=question.name, + value=suggestion.value, + source="suggestion", + record_id=record.id, + agent=suggestion.agent, + score=suggestion.score, + ) + ) + else: + cells.append(ProjectionCell(question_name=question.name, value=None, source=None)) +``` + +- [ ] **Step 4: Run to verify pass** + +Run: `cd extralit-server && uv run pytest tests/integration/contexts/v2/test_projection.py tests/integration/api/v2/test_projection.py -v` +Expected: ALL PASS. + +- [ ] **Step 5: Commit** + +```bash +git add extralit-server/src/extralit_server/api/schemas/v2/projection.py extralit-server/src/extralit_server/contexts/v2/projection.py extralit-server/tests/integration/contexts/v2/test_projection.py extralit-server/tests/integration/api/v2/test_projection.py +git commit -m "feat(server): enrich ProjectionCell with record_id/agent/score provenance" +``` + +--- + +### Task 3: Workspace-level denormalized projection context (server) + +**Files:** +- Modify: `extralit-server/src/extralit_server/api/schemas/v2/projection.py` (add 4 models) +- Modify: `extralit-server/src/extralit_server/contexts/v2/projection.py` (add `build_workspace_view` + helpers) +- Create: `extralit-server/tests/integration/contexts/v2/test_workspace_projection.py` + +**Interfaces:** +- Consumes: `Schema`, `V2Question` (`.columns` binding, `.type`), `V2Record` (`.reference`), `V2Suggestion`, `V2Response` ORM models; `ResponseStatus`, `QuestionType` enums. +- Produces: `async def build_workspace_view(db: AsyncSession, *, workspace_id: UUID, offset: int, limit: int) -> WorkspaceProjection` — Task 4's route calls exactly this. Pydantic models `WorkspaceProjectionColumn`, `WorkspaceProjectionCell`, `WorkspaceProjectionRow`, `WorkspaceProjection` (names Task 4 and gen:api rely on). +- Semantics locked here: `limit`/`offset` count **references** (not fan-out rows); cell coalesce = **latest submitted response by any user** (by `updated_at`) `??` suggestion `??` omitted; one **effective record** per (reference, schema) = latest `inserted_at`; table fan-out with **independent stacking** (row count = max fan-out, min 1) and **scalar repetition**; absent cells omitted from `cells`. + +- [ ] **Step 1: Add the Pydantic models** + +Append to `src/extralit_server/api/schemas/v2/projection.py`: + +```python +class WorkspaceProjectionColumn(BaseModel): + name: str # flat "Schema.question" / "Schema.question.subcol" (spec §3.1) + schema_id: UUID + schema_name: str + question_name: str + sub_column: str | None = None + dtype: str # the question type value; the grid treats it as informational + + +class WorkspaceProjectionCell(BaseModel): + value: Any | None = None + source: Literal["response", "suggestion"] + record_id: UUID + agent: str | None = None + score: float | list[float] | None = None + + +class WorkspaceProjectionRow(BaseModel): + reference: str + row_index: int + cells: dict[str, WorkspaceProjectionCell] # keyed by column name; absent cells omitted + + +class WorkspaceProjection(BaseModel): + columns: list[WorkspaceProjectionColumn] + rows: list[WorkspaceProjectionRow] + total_references: int +``` + +- [ ] **Step 2: Write the failing tests** + +Create `tests/integration/contexts/v2/test_workspace_projection.py`: + +```python +import pytest + +from extralit_server.contexts.v2 import projection as projection_ctx +from extralit_server.enums import QuestionType, ResponseStatus + +from tests.factories import ( + SchemaFactory, + SchemaVersionFactory, + UserFactory, + V2QuestionFactory, + V2RecordFactory, + V2ResponseFactory, + V2SuggestionFactory, + WorkspaceFactory, +) + +pytestmark = pytest.mark.asyncio + + +async def _make_schema(workspace, name: str): + schema = await SchemaFactory.create(workspace=workspace, name=name) + version = await SchemaVersionFactory.create(schema=schema) + return schema, version + + +async def _add_question(schema, name: str, *, qtype=QuestionType.text, columns=None): + return await V2QuestionFactory.create(schema=schema, name=name, type=qtype, columns=columns or [name]) + + +async def test_columns_manifest_covers_all_schemas_and_fans_out_table_bindings(db): + workspace = await WorkspaceFactory.create() + design, _ = await _make_schema(workspace, "Design") + outcomes, _ = await _make_schema(workspace, "Outcomes") + await _add_question(design, "type") + await _add_question(outcomes, "results", qtype=QuestionType.table, columns=["value", "unit"]) + + view = await projection_ctx.build_workspace_view(db, workspace_id=workspace.id, offset=0, limit=50) + + names = [c.name for c in view.columns] + assert names == ["Design.type", "Outcomes.results.value", "Outcomes.results.unit"] + table_col = view.columns[1] + assert table_col.schema_name == "Outcomes" + assert table_col.question_name == "results" + assert table_col.sub_column == "value" + assert table_col.dtype == "table" + + +async def test_row_universe_is_union_of_references_with_coverage_gaps(db): + workspace = await WorkspaceFactory.create() + design, design_v = await _make_schema(workspace, "Design") + outcomes, outcomes_v = await _make_schema(workspace, "Outcomes") + dq = await _add_question(design, "type") + await _add_question(outcomes, "summary") + rec = await V2RecordFactory.create(version=design_v, reference="10.1/a") + await V2SuggestionFactory.create(record=rec, question=dq, value="RCT") + await V2RecordFactory.create(version=outcomes_v, reference="10.1/b") + + view = await projection_ctx.build_workspace_view(db, workspace_id=workspace.id, offset=0, limit=50) + + assert view.total_references == 2 + assert [(r.reference, r.row_index) for r in view.rows] == [("10.1/a", 0), ("10.1/b", 0)] + row_a, row_b = view.rows + assert row_a.cells["Design.type"].value == "RCT" + assert "Outcomes.summary" not in row_a.cells # no Outcomes record: coverage gap, cell omitted + assert row_b.cells == {} # record exists but neither response nor suggestion + + +async def test_latest_submitted_response_any_user_beats_suggestion(db): + workspace = await WorkspaceFactory.create() + schema, version = await _make_schema(workspace, "Design") + q = await _add_question(schema, "type") + rec = await V2RecordFactory.create(version=version, reference="10.1/a") + await V2SuggestionFactory.create(record=rec, question=q, value="cohort", agent="gpt-x", score=0.9) + user1 = await UserFactory.create() + user2 = await UserFactory.create() + await V2ResponseFactory.create( + record=rec, user=user1, values={"type": {"value": "RCT-old"}}, status=ResponseStatus.submitted + ) + await V2ResponseFactory.create( + record=rec, user=user2, values={"type": {"value": "RCT"}}, status=ResponseStatus.submitted + ) + + view = await projection_ctx.build_workspace_view(db, workspace_id=workspace.id, offset=0, limit=50) + + cell = view.rows[0].cells["Design.type"] + assert cell.value == "RCT" # later updated_at wins across users + assert cell.source == "response" + assert cell.record_id == rec.id + assert cell.agent is None and cell.score is None + + +async def test_draft_responses_never_appear(db): + workspace = await WorkspaceFactory.create() + schema, version = await _make_schema(workspace, "Design") + q = await _add_question(schema, "type") + rec = await V2RecordFactory.create(version=version, reference="10.1/a") + await V2SuggestionFactory.create(record=rec, question=q, value="cohort", agent="gpt-x", score=0.9) + user = await UserFactory.create() + await V2ResponseFactory.create( + record=rec, user=user, values={"type": {"value": "draft-val"}}, status=ResponseStatus.draft + ) + + view = await projection_ctx.build_workspace_view(db, workspace_id=workspace.id, offset=0, limit=50) + + cell = view.rows[0].cells["Design.type"] + assert cell.value == "cohort" + assert cell.source == "suggestion" + assert cell.agent == "gpt-x" + assert cell.score == 0.9 + + +async def test_table_fanout_independent_stacking_and_scalar_repetition(db): + workspace = await WorkspaceFactory.create() + schema, version = await _make_schema(workspace, "Outcomes") + scalar_q = await _add_question(schema, "design") + t1 = await _add_question(schema, "results", qtype=QuestionType.table, columns=["value", "unit"]) + t2 = await _add_question(schema, "arms", qtype=QuestionType.table, columns=["arm"]) + rec = await V2RecordFactory.create(version=version, reference="10.1/a") + await V2SuggestionFactory.create(record=rec, question=scalar_q, value="RCT") + await V2SuggestionFactory.create( + record=rec, + question=t1, + value=[{"value": "12%", "unit": "pct"}, {"value": "8%", "unit": "pct"}, {"value": "3%"}], + ) + await V2SuggestionFactory.create(record=rec, question=t2, value=[{"arm": "control"}, {"arm": "treated"}]) + + view = await projection_ctx.build_workspace_view(db, workspace_id=workspace.id, offset=0, limit=50) + + assert view.total_references == 1 + assert len(view.rows) == 3 # max(3, 2), NOT 3*2 (no cartesian product) + assert [r.row_index for r in view.rows] == [0, 1, 2] + # scalars repeat on every fan-out row (true denormalized rows) + assert all(r.cells["Outcomes.design"].value == "RCT" for r in view.rows) + assert [r.cells["Outcomes.results.value"].value for r in view.rows] == ["12%", "8%", "3%"] + # shorter table just ends (independent stacking): row 2 has no arms cell + assert [r.cells.get("Outcomes.arms.arm") and r.cells["Outcomes.arms.arm"].value for r in view.rows] == [ + "control", + "treated", + None, + ] + # missing sub-key on a row dict is omitted + assert "Outcomes.results.unit" not in view.rows[2].cells + + +async def test_single_dict_table_value_is_one_row(db): + workspace = await WorkspaceFactory.create() + schema, version = await _make_schema(workspace, "Outcomes") + t = await _add_question(schema, "results", qtype=QuestionType.table, columns=["value"]) + rec = await V2RecordFactory.create(version=version, reference="10.1/a") + await V2SuggestionFactory.create(record=rec, question=t, value={"value": "12%"}) + + view = await projection_ctx.build_workspace_view(db, workspace_id=workspace.id, offset=0, limit=50) + + assert len(view.rows) == 1 + assert view.rows[0].cells["Outcomes.results.value"].value == "12%" + + +async def test_effective_record_is_latest_inserted_per_reference_schema(db): + workspace = await WorkspaceFactory.create() + schema, version = await _make_schema(workspace, "Design") + q = await _add_question(schema, "type") + old = await V2RecordFactory.create(version=version, reference="10.1/a") + new = await V2RecordFactory.create(version=version, reference="10.1/a") + await V2SuggestionFactory.create(record=old, question=q, value="old") + await V2SuggestionFactory.create(record=new, question=q, value="new") + + view = await projection_ctx.build_workspace_view(db, workspace_id=workspace.id, offset=0, limit=50) + + assert len(view.rows) == 1 + assert view.rows[0].cells["Design.type"].value == "new" + assert view.rows[0].cells["Design.type"].record_id == new.id + + +async def test_pagination_counts_references_not_rows(db): + workspace = await WorkspaceFactory.create() + schema, version = await _make_schema(workspace, "Design") + await _add_question(schema, "type") + for i in range(5): + await V2RecordFactory.create(version=version, reference=f"10.1/{i}") + + page = await projection_ctx.build_workspace_view(db, workspace_id=workspace.id, offset=2, limit=2) + + assert page.total_references == 5 + assert [r.reference for r in page.rows] == ["10.1/2", "10.1/3"] # ordered by reference + + +async def test_query_count_is_constant_regardless_of_reference_count(db, monkeypatch): + workspace = await WorkspaceFactory.create() + schema, version = await _make_schema(workspace, "Design") + q = await _add_question(schema, "type") + for i in range(6): + rec = await V2RecordFactory.create(version=version, reference=f"10.1/{i}") + await V2SuggestionFactory.create(record=rec, question=q, value=f"v{i}") + + executed: list[object] = [] + original_execute = db.execute + + async def counting_execute(*args, **kwargs): + executed.append(args[0]) + return await original_execute(*args, **kwargs) + + monkeypatch.setattr(db, "execute", counting_execute) + await projection_ctx.build_workspace_view(db, workspace_id=workspace.id, offset=0, limit=50) + # schemas, questions, ref-count, ref-page, records, suggestions, responses => 7 max + assert len(executed) <= 7, f"N+1 regression: {len(executed)} statements" +``` + +Note: if `V2ResponseFactory`/`V2QuestionFactory` kwargs differ (check `tests/factories.py` lines ~655–764), adapt the factory calls — the factories use custom async `_create` that awaits SubFactory coroutines; passing explicit `record=`/`question=`/`user=`/`schema=`/`version=` instances is the established pattern in `tests/integration/contexts/v2/test_projection.py`. + +- [ ] **Step 3: Run to verify failure** + +Run: `cd extralit-server && uv run pytest tests/integration/contexts/v2/test_workspace_projection.py -v` +Expected: FAIL — `module … has no attribute 'build_workspace_view'`. + +- [ ] **Step 4: Implement** + +In `src/extralit_server/contexts/v2/projection.py`, extend the imports (keep existing ones): + +```python +from sqlalchemy import func, select + +from extralit_server.api.schemas.v2.projection import ( + ProjectionCell, + ProjectionRecord, + ProjectionView, + WorkspaceProjection, + WorkspaceProjectionCell, + WorkspaceProjectionColumn, + WorkspaceProjectionRow, +) +from extralit_server.enums import QuestionType, ResponseStatus +from extralit_server.models.v2 import Schema, V2Question, V2Record, V2Response, V2Suggestion +``` + +Append: + +```python +def _build_columns( + schemas: list[Schema], questions_by_schema: dict, +) -> list[WorkspaceProjectionColumn]: + columns: list[WorkspaceProjectionColumn] = [] + for schema in schemas: + for question in questions_by_schema.get(schema.id, []): + if question.type == QuestionType.table: + # sub-columns are the question's `columns` binding (spec §3.4) + for sub in question.columns: + columns.append( + WorkspaceProjectionColumn( + name=f"{schema.name}.{question.name}.{sub}", + schema_id=schema.id, + schema_name=schema.name, + question_name=question.name, + sub_column=sub, + dtype=question.type.value, + ) + ) + else: + columns.append( + WorkspaceProjectionColumn( + name=f"{schema.name}.{question.name}", + schema_id=schema.id, + schema_name=schema.name, + question_name=question.name, + sub_column=None, + dtype=question.type.value, + ) + ) + return columns + + +def _resolve(record, question, suggestions, responses): + """Coalesce (spec §3.1): latest submitted response (any user) ?? suggestion ?? None. + Returns (value, source, agent, score) or None. Drafts never reach `responses`.""" + wrapped = responses.get(record.id, {}).get(question.name) + if wrapped is not None: + return (wrapped.get("value"), "response", None, None) + suggestion = suggestions.get((record.id, question.id)) + if suggestion is not None: + return (suggestion.value, "suggestion", suggestion.agent, suggestion.score) + return None + + +def _table_rows(value) -> list[dict]: + # §3.4 normalization: bare dict is the 1-row case; list[dict] is N rows. + if isinstance(value, list): + return [row for row in value if isinstance(row, dict)] + if isinstance(value, dict): + return [value] + return [] + + +def _denormalize_reference( + reference: str, + *, + schemas: list[Schema], + questions_by_schema: dict, + effective: dict, + suggestions: dict, + responses: dict, +) -> list[WorkspaceProjectionRow]: + scalar_cells: dict[str, WorkspaceProjectionCell] = {} + table_values: list[tuple] = [] # (schema_name, question, rows, record, (source, agent, score)) + fan_out = 1 + for schema in schemas: + record = effective.get((reference, schema.id)) + if record is None: + continue # coverage gap: this schema's columns stay empty for the reference + for question in questions_by_schema.get(schema.id, []): + resolved = _resolve(record, question, suggestions, responses) + if resolved is None: + continue # absent cells are omitted; the client renders them as null + value, source, agent, score = resolved + if question.type == QuestionType.table: + rows = _table_rows(value) + if not rows: + continue + fan_out = max(fan_out, len(rows)) + table_values.append((schema.name, question, rows, record, (source, agent, score))) + elif value is not None: + scalar_cells[f"{schema.name}.{question.name}"] = WorkspaceProjectionCell( + value=value, source=source, record_id=record.id, agent=agent, score=score + ) + + out: list[WorkspaceProjectionRow] = [] + for row_index in range(fan_out): + cells = dict(scalar_cells) # scalars repeat on every fan-out row (denormalized, §3.1) + for schema_name, question, rows, record, (source, agent, score) in table_values: + if row_index >= len(rows): + continue # independent stacking: shorter tables just end, no fabricated pairing + for sub in question.columns: + sub_value = rows[row_index].get(sub) + if sub_value is None: + continue + cells[f"{schema_name}.{question.name}.{sub}"] = WorkspaceProjectionCell( + value=sub_value, source=source, record_id=record.id, agent=agent, score=score + ) + out.append(WorkspaceProjectionRow(reference=reference, row_index=row_index, cells=cells)) + return out + + +async def build_workspace_view( + db: AsyncSession, *, workspace_id: UUID, offset: int, limit: int +) -> WorkspaceProjection: + """Workspace-level denormalized projection (spec §3.2). Batched: a fixed number of + queries regardless of reference/record count. `offset`/`limit` count references.""" + schemas = ( + (await db.execute(select(Schema).where(Schema.workspace_id == workspace_id).order_by(Schema.name))) + .scalars() + .all() + ) + schema_ids = [s.id for s in schemas] + + questions_by_schema: dict[UUID, list[V2Question]] = {} + if schema_ids: + question_rows = ( + ( + await db.execute( + select(V2Question) + .where(V2Question.schema_id.in_(schema_ids)) + .order_by(V2Question.inserted_at) + ) + ) + .scalars() + .all() + ) + for question in question_rows: + questions_by_schema.setdefault(question.schema_id, []).append(question) + + columns = _build_columns(schemas, questions_by_schema) + + if not schema_ids: + return WorkspaceProjection(columns=columns, rows=[], total_references=0) + + # Row universe: every distinct reference across the workspace's records (§3.1). + ref_query = select(V2Record.reference).where(V2Record.schema_id.in_(schema_ids)).distinct() + total_references = ( + await db.execute(select(func.count()).select_from(ref_query.subquery())) + ).scalar_one() + references = ( + (await db.execute(ref_query.order_by(V2Record.reference).offset(offset).limit(limit))) + .scalars() + .all() + ) + if not references: + return WorkspaceProjection(columns=columns, rows=[], total_references=total_references) + + record_rows = ( + ( + await db.execute( + select(V2Record) + .where(V2Record.schema_id.in_(schema_ids), V2Record.reference.in_(references)) + .order_by(V2Record.inserted_at) + ) + ) + .scalars() + .all() + ) + # One effective record per (reference, schema): latest inserted_at wins (§3.1). + effective: dict[tuple[str, UUID], V2Record] = {} + for record in record_rows: + effective[(record.reference, record.schema_id)] = record + record_ids = [r.id for r in effective.values()] + + suggestions: dict[tuple[UUID, UUID], V2Suggestion] = {} + responses: dict[UUID, dict] = {} + if record_ids: + sugg_rows = ( + (await db.execute(select(V2Suggestion).where(V2Suggestion.record_id.in_(record_ids)))) + .scalars() + .all() + ) + suggestions = {(s.record_id, s.question_id): s for s in sugg_rows} + # Latest submitted per record across ALL users — endpoint is user-agnostic and + # cacheable (§3.1); ordering by updated_at lets the dict overwrite older entries. + resp_rows = ( + ( + await db.execute( + select(V2Response) + .where( + V2Response.record_id.in_(record_ids), + V2Response.status == ResponseStatus.submitted, + ) + .order_by(V2Response.updated_at) + ) + ) + .scalars() + .all() + ) + for resp in resp_rows: + responses[resp.record_id] = resp.values or {} + + rows: list[WorkspaceProjectionRow] = [] + for reference in references: + rows.extend( + _denormalize_reference( + reference, + schemas=schemas, + questions_by_schema=questions_by_schema, + effective=effective, + suggestions=suggestions, + responses=responses, + ) + ) + return WorkspaceProjection(columns=columns, rows=rows, total_references=total_references) +``` + +Note: `build_reference_view`'s existing imports already cover most of this; merge rather than duplicate import lines. `AsyncSession`/`UUID` are already imported at the top of the module. + +- [ ] **Step 5: Run to verify pass** + +Run: `cd extralit-server && uv run pytest tests/integration/contexts/v2/test_workspace_projection.py tests/integration/contexts/v2/test_projection.py -v` +Expected: ALL PASS (including the ≤7-statement query-count guard). + +- [ ] **Step 6: Commit** + +```bash +git add extralit-server/src/extralit_server/api/schemas/v2/projection.py extralit-server/src/extralit_server/contexts/v2/projection.py extralit-server/tests/integration/contexts/v2/test_workspace_projection.py +git commit -m "feat(server): workspace-level denormalized projection with table fan-out" +``` + +--- + +### Task 4: `GET /api/v2/projection` route (server) + +**Files:** +- Modify: `extralit-server/src/extralit_server/api/v2/projection.py` +- Test: `extralit-server/tests/integration/api/v2/test_projection.py` + +**Interfaces:** +- Consumes: `build_workspace_view` (Task 3), existing `auth`/`authorize`/`SchemaPolicy.list` pattern. +- Produces: `GET /api/v2/projection?workspace_id=…&offset=0&limit=50` → `WorkspaceProjection` JSON (`columns`, `rows`, `total_references`). `limit` ∈ [1, 100], default 50. This is the contract the frontend types against after gen:api. + +- [ ] **Step 1: Write the failing tests** + +Append to `tests/integration/api/v2/test_projection.py` (reuse its existing `_schema_with_question` helper and imports; add `V2ResponseFactory`, `UserFactory` imports if not present): + +```python +async def test_workspace_projection_returns_manifest_rows_and_total(async_client, owner_auth_header): + workspace = await WorkspaceFactory.create() + schema, version, q = await _schema_with_question(workspace) + record = await V2RecordFactory.create(version=version, reference="doc-1") + await V2SuggestionFactory.create(record=record, question=q, value="flu", agent="gpt-x", score=0.92) + + resp = await async_client.get( + f"/api/v2/projection?workspace_id={workspace.id}", headers=owner_auth_header + ) + + assert resp.status_code == 200, resp.text + body = resp.json() + assert body["total_references"] == 1 + [column] = [c for c in body["columns"] if c["question_name"] == q.name] + assert column["schema_id"] == str(schema.id) + assert column["sub_column"] is None + [row] = body["rows"] + assert row["reference"] == "doc-1" + assert row["row_index"] == 0 + cell = row["cells"][column["name"]] + assert cell == { + "value": "flu", + "source": "suggestion", + "record_id": str(record.id), + "agent": "gpt-x", + "score": 0.92, + } + + +async def test_workspace_projection_paginates_references(async_client, owner_auth_header): + workspace = await WorkspaceFactory.create() + schema, version, q = await _schema_with_question(workspace) + for i in range(3): + await V2RecordFactory.create(version=version, reference=f"doc-{i}") + + resp = await async_client.get( + f"/api/v2/projection?workspace_id={workspace.id}&offset=1&limit=1", headers=owner_auth_header + ) + + assert resp.status_code == 200, resp.text + body = resp.json() + assert body["total_references"] == 3 + assert [r["reference"] for r in body["rows"]] == ["doc-1"] + + +async def test_workspace_projection_rejects_limit_over_100(async_client, owner_auth_header): + workspace = await WorkspaceFactory.create() + resp = await async_client.get( + f"/api/v2/projection?workspace_id={workspace.id}&limit=101", headers=owner_auth_header + ) + assert resp.status_code == 422 +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cd extralit-server && uv run pytest tests/integration/api/v2/test_projection.py -v` +Expected: new tests FAIL with 404 (route absent); Task 2's tests still PASS. + +- [ ] **Step 3: Implement** + +In `src/extralit_server/api/v2/projection.py`, import `WorkspaceProjection` alongside `ProjectionView`, and add ABOVE the existing per-reference route (order is not load-bearing — the paths don't overlap — but keeps the file readable): + +```python +@router.get("/projection", response_model=WorkspaceProjection) +async def get_workspace_projection( + *, + db: Annotated[AsyncSession, Depends(get_async_db)], + current_user: Annotated[User, Security(auth.get_current_user)], + workspace_id: Annotated[UUID, Query(description="Workspace to scope the view (required)")], + offset: Annotated[int, Query(ge=0, description="Reference offset (not fan-out rows)")] = 0, + limit: Annotated[int, Query(ge=1, le=100, description="References per page")] = 50, +): + await authorize(current_user, SchemaPolicy.list(workspace_id)) + return await projection_ctx.build_workspace_view( + db, workspace_id=workspace_id, offset=offset, limit=limit + ) +``` + +- [ ] **Step 4: Run to verify pass** + +Run: `cd extralit-server && uv run pytest tests/integration/api/v2/test_projection.py -v` +Expected: ALL PASS. + +- [ ] **Step 5: Commit** + +```bash +git add extralit-server/src/extralit_server/api/v2/projection.py extralit-server/tests/integration/api/v2/test_projection.py +git commit -m "feat(server): GET /api/v2/projection workspace endpoint" +``` + +--- + +### Task 5: gen:api + domain types + `ProjectionRepository.getWorkspaceProjection` (frontend) + +**Files:** +- Create: `extralit-frontend/v2/domain/entities/projection/WorkspaceProjection.ts` +- Modify: `extralit-frontend/v2/infrastructure/repositories/ProjectionRepository.ts` +- Modify (generated): `extralit-frontend/v2/infrastructure/api/openapi.json`, `extralit-frontend/v2/infrastructure/api/generated/v2-api.ts` +- Test: `extralit-frontend/v2/infrastructure/repositories/ProjectionRepository.test.ts` + +**Interfaces:** +- Consumes: generated `components["schemas"]["WorkspaceProjection"]` from Task 4's contract. +- Produces (used by Tasks 6–10): + - `ProjectionColumn { name; schemaId; schemaName; questionName; subColumn: string | null; dtype: string }` + - `ProjectionGridCell { value: unknown; source: "response" | "suggestion"; recordId: string; agent: string | null; score: number | number[] | null }` + - `ProjectionGridRow { reference: string; rowIndex: number; cells: Record }` + - `class WorkspaceProjection { columns: ProjectionColumn[]; rows: ProjectionGridRow[]; totalReferences: number }` + - `ProjectionRepository.getWorkspaceProjection(workspaceId: string, offset?: number, limit?: number): Promise` where `WorkspaceProjectionPageDto { columns: ProjectionColumn[]; rows: ProjectionGridRow[]; totalReferences: number }` + +- [ ] **Step 1: Regenerate the API contract** + +```bash +cd extralit-frontend && npm run gen:api +git diff --stat v2/infrastructure/api +``` +Expected: both `openapi.json` and `generated/v2-api.ts` change; `v2-api.ts` gains `WorkspaceProjection`, `WorkspaceProjectionColumn`, `WorkspaceProjectionCell`, `WorkspaceProjectionRow` schemas plus `record_id`/`agent`/`score` on `ProjectionCell`. + +- [ ] **Step 2: Create the domain types** + +Create `v2/domain/entities/projection/WorkspaceProjection.ts`: + +```ts +export interface ProjectionColumn { + name: string; // flat "Schema.question" / "Schema.question.subcol" + schemaId: string; + schemaName: string; + questionName: string; + subColumn: string | null; + dtype: string; +} + +export interface ProjectionGridCell { + value: unknown; + source: "response" | "suggestion"; + recordId: string; + agent: string | null; + score: number | number[] | null; +} + +export interface ProjectionGridRow { + reference: string; + rowIndex: number; + cells: Record; +} + +export class WorkspaceProjection { + constructor( + public readonly columns: ProjectionColumn[], + public readonly rows: ProjectionGridRow[], + public readonly totalReferences: number + ) {} +} +``` + +- [ ] **Step 3: Write the failing repository test** + +Append to `v2/infrastructure/repositories/ProjectionRepository.test.ts` (keep the existing per-reference test untouched): + +```ts +const BACKEND_WORKSPACE = { + columns: [ + { + name: "Design.type", + schema_id: "s-1", + schema_name: "Design", + question_name: "type", + sub_column: null, + dtype: "text", + }, + ], + rows: [ + { + reference: "10.1000/j.x", + row_index: 0, + cells: { + "Design.type": { value: "RCT", source: "response", record_id: "r-1", agent: null, score: null }, + }, + }, + ], + total_references: 213, +}; + +describe("getWorkspaceProjection", () => { + it("pages the workspace projection and maps snake_case to the domain shape", async () => { + const axios = { get: vi.fn(async () => ({ data: BACKEND_WORKSPACE })) }; + const page = await new ProjectionRepository(axios as never).getWorkspaceProjection("w-1", 50, 25); + + expect(axios.get).toHaveBeenCalledWith("/v2/projection", { + params: { workspace_id: "w-1", offset: 50, limit: 25 }, + }); + expect(page.totalReferences).toBe(213); + expect(page.columns[0]).toEqual({ + name: "Design.type", + schemaId: "s-1", + schemaName: "Design", + questionName: "type", + subColumn: null, + dtype: "text", + }); + expect(page.rows[0].reference).toBe("10.1000/j.x"); + expect(page.rows[0].rowIndex).toBe(0); + expect(page.rows[0].cells["Design.type"]).toEqual({ + value: "RCT", + source: "response", + recordId: "r-1", + agent: null, + score: null, + }); + }); + + it("defaults to offset 0, limit 50", async () => { + const axios = { get: vi.fn(async () => ({ data: BACKEND_WORKSPACE })) }; + await new ProjectionRepository(axios as never).getWorkspaceProjection("w-1"); + expect(axios.get).toHaveBeenCalledWith("/v2/projection", { + params: { workspace_id: "w-1", offset: 0, limit: 50 }, + }); + }); +}); +``` + +- [ ] **Step 4: Run to verify failure** + +Run: `cd extralit-frontend && npx vitest run v2/infrastructure/repositories/ProjectionRepository.test.ts` +Expected: FAIL — `getWorkspaceProjection is not a function`. + +- [ ] **Step 5: Implement the repository method** + +In `v2/infrastructure/repositories/ProjectionRepository.ts` add (keeping the existing `getProjection` untouched): + +```ts +import { + type ProjectionColumn, + type ProjectionGridCell, + type ProjectionGridRow, +} from "~/v2/domain/entities/projection/WorkspaceProjection"; + +type BackendWorkspaceProjection = components["schemas"]["WorkspaceProjection"]; + +export interface WorkspaceProjectionPageDto { + columns: ProjectionColumn[]; + rows: ProjectionGridRow[]; + totalReferences: number; +} +``` + +and inside the class: + +```ts + async getWorkspaceProjection( + workspaceId: string, + offset = 0, + limit = 50 + ): Promise { + const { data } = await this.axios.get("/v2/projection", { + params: { workspace_id: workspaceId, offset, limit }, + }); + return { + totalReferences: data.total_references, + columns: data.columns.map((c) => ({ + name: c.name, + schemaId: c.schema_id, + schemaName: c.schema_name, + questionName: c.question_name, + subColumn: c.sub_column ?? null, + dtype: c.dtype, + })), + rows: data.rows.map((r) => ({ + reference: r.reference, + rowIndex: r.row_index, + cells: Object.fromEntries( + Object.entries(r.cells).map(([name, cell]) => [ + name, + { + value: cell.value ?? null, + source: cell.source, + recordId: cell.record_id, + agent: cell.agent ?? null, + score: cell.score ?? null, + } satisfies ProjectionGridCell, + ]) + ), + })), + }; + } +``` + +- [ ] **Step 6: Run to verify pass** + +Run: `cd extralit-frontend && npx vitest run v2/infrastructure/repositories/ProjectionRepository.test.ts` +Expected: ALL PASS (old + new). + +- [ ] **Step 7: Commit** + +```bash +git add extralit-frontend/v2/infrastructure/api extralit-frontend/v2/domain/entities/projection/WorkspaceProjection.ts extralit-frontend/v2/infrastructure/repositories/ProjectionRepository.ts extralit-frontend/v2/infrastructure/repositories/ProjectionRepository.test.ts +git commit -m "feat(v2-ui): workspace projection contract, domain types, repository method" +``` + +--- + +### Task 6: Grid adapter — pure denormalized-rows → Perspective mapping (frontend) + +**Files:** +- Create: `extralit-frontend/v2/domain/entities/projection/grid-adapter.ts` +- Test: `extralit-frontend/v2/domain/entities/projection/grid-adapter.test.ts` + +**Interfaces:** +- Consumes: `WorkspaceProjection`, `ProjectionGridCell` (Task 5). +- Produces (used by Tasks 9–10): + - `REFERENCE_COLUMN = "reference"` + - `toPerspectiveData(projection: WorkspaceProjection): Record[]` — one flat object per row, EVERY manifest column present (`null` when absent) so the inferred Perspective schema is stable + - `cellAt(projection: WorkspaceProjection, rowIndex: number, columnName: string): ProjectionGridCell | null` + - `bandParity(projection: WorkspaceProjection): number[]` — 0/1 per row, flips when `reference` changes + - `ANNOTATION_CELL_LINKS_ENABLED = false` (spec §3.3 guard) + - `buildAnnotationUrl(schemaId: string, reference: string): string` → `/dataset/{schemaId}/annotation-mode?_search={encoded reference}` + +- [ ] **Step 1: Write the failing tests** + +Create `v2/domain/entities/projection/grid-adapter.test.ts`: + +```ts +import { describe, expect, it } from "vitest"; +import { + ANNOTATION_CELL_LINKS_ENABLED, + bandParity, + buildAnnotationUrl, + cellAt, + REFERENCE_COLUMN, + toPerspectiveData, +} from "./grid-adapter"; +import { WorkspaceProjection, type ProjectionGridCell } from "./WorkspaceProjection"; + +const cell = (value: unknown): ProjectionGridCell => ({ + value, + source: "suggestion", + recordId: "r-1", + agent: "gpt-x", + score: 0.9, +}); + +const COLUMNS = [ + { name: "Design.type", schemaId: "s-1", schemaName: "Design", questionName: "type", subColumn: null, dtype: "text" }, + { + name: "Outcomes.results.value", + schemaId: "s-2", + schemaName: "Outcomes", + questionName: "results", + subColumn: "value", + dtype: "table", + }, +]; + +const PROJECTION = new WorkspaceProjection( + COLUMNS, + [ + { reference: "10.1/a", rowIndex: 0, cells: { "Design.type": cell("RCT") } }, + { reference: "10.1/a", rowIndex: 1, cells: { "Outcomes.results.value": cell("12%") } }, + { reference: "10.1/b", rowIndex: 0, cells: {} }, + ], + 2 +); + +describe("toPerspectiveData", () => { + it("emits one flat record per row with every manifest column (null when absent)", () => { + expect(toPerspectiveData(PROJECTION)).toEqual([ + { [REFERENCE_COLUMN]: "10.1/a", "Design.type": "RCT", "Outcomes.results.value": null }, + { [REFERENCE_COLUMN]: "10.1/a", "Design.type": null, "Outcomes.results.value": "12%" }, + { [REFERENCE_COLUMN]: "10.1/b", "Design.type": null, "Outcomes.results.value": null }, + ]); + }); +}); + +describe("cellAt", () => { + it("returns the enriched cell for the loaded row order", () => { + expect(cellAt(PROJECTION, 0, "Design.type")?.recordId).toBe("r-1"); + }); + + it("returns null for empty cells and out-of-range rows", () => { + expect(cellAt(PROJECTION, 2, "Design.type")).toBeNull(); + expect(cellAt(PROJECTION, 99, "Design.type")).toBeNull(); + }); +}); + +describe("bandParity", () => { + it("flips parity when the reference changes, not per row", () => { + expect(bandParity(PROJECTION)).toEqual([0, 0, 1]); + }); +}); + +describe("annotation URL contract (spec §3.3)", () => { + it("stays guarded off in this build", () => { + expect(ANNOTATION_CELL_LINKS_ENABLED).toBe(false); + }); + + it("puts the schema id in the dataset slot and encodes the reference", () => { + expect(buildAnnotationUrl("s-1", "10.1/a b")).toBe("/dataset/s-1/annotation-mode?_search=10.1%2Fa%20b"); + }); +}); +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cd extralit-frontend && npx vitest run v2/domain/entities/projection/grid-adapter.test.ts` +Expected: FAIL — module not found. + +- [ ] **Step 3: Implement** + +Create `v2/domain/entities/projection/grid-adapter.ts`: + +```ts +import { type ProjectionGridCell, type WorkspaceProjection } from "./WorkspaceProjection"; + +export const REFERENCE_COLUMN = "reference"; + +// Flat denormalized records for `client.table(data)`. Every manifest column is present +// on every row (null when absent) so Perspective infers a stable schema. +export const toPerspectiveData = (projection: WorkspaceProjection): Record[] => + projection.rows.map((row) => { + const flat: Record = { [REFERENCE_COLUMN]: row.reference }; + for (const column of projection.columns) { + flat[column.name] = row.cells[column.name]?.value ?? null; + } + return flat; + }); + +// Click plumbing (spec §3.3): rows load in projection order and the grid is static +// (no sort/filter), so a datagrid row index maps 1:1 onto projection.rows. +export const cellAt = ( + projection: WorkspaceProjection, + rowIndex: number, + columnName: string +): ProjectionGridCell | null => projection.rows[rowIndex]?.cells[columnName] ?? null; + +// Row-banding by reference (spec §3.1) — Perspective is flat-columnar and cannot merge +// cells, so visual grouping is alternating bands per reference block. +export const bandParity = (projection: WorkspaceProjection): number[] => { + let parity = 0; + let previous: string | null = null; + return projection.rows.map((row) => { + if (previous !== null && row.reference !== previous) parity ^= 1; + previous = row.reference; + return parity; + }); +}; + +// Future annotation URL contract (spec §3.3): schema id occupies the dataset slot. +// Guarded OFF until annotation-mode resolves v2 schema ids (ledger §5). +export const ANNOTATION_CELL_LINKS_ENABLED = false; + +export const buildAnnotationUrl = (schemaId: string, reference: string): string => + `/dataset/${schemaId}/annotation-mode?_search=${encodeURIComponent(reference)}`; +``` + +- [ ] **Step 4: Run to verify pass** + +Run: `cd extralit-frontend && npx vitest run v2/domain/entities/projection/grid-adapter.test.ts` +Expected: ALL PASS. + +- [ ] **Step 5: Commit** + +```bash +git add extralit-frontend/v2/domain/entities/projection/grid-adapter.ts extralit-frontend/v2/domain/entities/projection/grid-adapter.test.ts +git commit -m "feat(v2-ui): pure grid adapter — flat rows, banding, click-URL contract" +``` + +--- + +### Task 7: `GetWorkspaceProjectionUseCase` + `ExtractionsStorage` + DI (frontend) + +**Files:** +- Create: `extralit-frontend/v2/infrastructure/storage/ExtractionsStorage.ts` +- Create: `extralit-frontend/v2/domain/usecases/get-workspace-projection-use-case.ts` +- Modify: `extralit-frontend/v2/di/di.ts` +- Test: `extralit-frontend/v2/domain/usecases/get-workspace-projection-use-case.test.ts` + +**Interfaces:** +- Consumes: `ProjectionRepository.getWorkspaceProjection` (Task 5), `useStoreFor` from `@/v1/store/create`. +- Produces: `GetWorkspaceProjectionUseCase.execute(workspaceId: string): Promise` (pages through ALL references at `PROJECTION_PAGE_SIZE = 100` per call, aggregates rows into ONE `WorkspaceProjection`, saves to storage); `useExtractions()` store with `saveProjection(projection)` and `get(): Extractions` where `Extractions.projection: WorkspaceProjection | null`. Task 10's view-model resolves the use-case via `useResolve`. + +- [ ] **Step 1: Create the storage** + +Create `v2/infrastructure/storage/ExtractionsStorage.ts`: + +```ts +import { useStoreFor } from "@/v1/store/create"; +import { WorkspaceProjection } from "~/v2/domain/entities/projection/WorkspaceProjection"; + +// Class name is the Pinia store key — must stay unique vs every v1/v2 useStoreFor class. +class Extractions { + constructor(public readonly projection: WorkspaceProjection | null = null) {} +} + +interface IExtractionsStorage { + saveProjection(projection: WorkspaceProjection): void; +} + +const useStoreForExtractions = useStoreFor(Extractions); + +export const useExtractions = () => { + const store = useStoreForExtractions(); + + const saveProjection = (projection: WorkspaceProjection) => { + store.save(new Extractions(projection)); + }; + + return { ...store, saveProjection }; +}; +``` + +- [ ] **Step 2: Write the failing use-case test** + +Create `v2/domain/usecases/get-workspace-projection-use-case.test.ts`: + +```ts +import { createPinia, setActivePinia } from "pinia"; +import { beforeEach, describe, expect, it, vi } from "vitest"; +import { GetWorkspaceProjectionUseCase, PROJECTION_PAGE_SIZE } from "./get-workspace-projection-use-case"; +import { useExtractions } from "~/v2/infrastructure/storage/ExtractionsStorage"; + +const COLUMN = { + name: "Design.type", + schemaId: "s-1", + schemaName: "Design", + questionName: "type", + subColumn: null, + dtype: "text", +}; + +const row = (reference: string) => ({ reference, rowIndex: 0, cells: {} }); + +describe("GetWorkspaceProjectionUseCase", () => { + beforeEach(() => setActivePinia(createPinia())); + + it("loads every page into one projection and saves it to storage", async () => { + const repository = { + getWorkspaceProjection: vi + .fn() + .mockResolvedValueOnce({ columns: [COLUMN], rows: [row("10.1/a")], totalReferences: 150 }) + .mockResolvedValueOnce({ columns: [COLUMN], rows: [row("10.1/b")], totalReferences: 150 }), + }; + const useCase = new GetWorkspaceProjectionUseCase(repository as never, useExtractions()); + + const projection = await useCase.execute("w-1"); + + expect(repository.getWorkspaceProjection).toHaveBeenNthCalledWith(1, "w-1", 0, PROJECTION_PAGE_SIZE); + expect(repository.getWorkspaceProjection).toHaveBeenNthCalledWith(2, "w-1", 100, PROJECTION_PAGE_SIZE); + expect(repository.getWorkspaceProjection).toHaveBeenCalledTimes(2); + expect(projection.rows.map((r) => r.reference)).toEqual(["10.1/a", "10.1/b"]); + expect(projection.totalReferences).toBe(150); + expect(useExtractions().get().projection).toEqual(projection); + }); + + it("makes a single call when everything fits in one page", async () => { + const repository = { + getWorkspaceProjection: vi + .fn() + .mockResolvedValue({ columns: [COLUMN], rows: [row("10.1/a")], totalReferences: 1 }), + }; + const useCase = new GetWorkspaceProjectionUseCase(repository as never, useExtractions()); + await useCase.execute("w-1"); + expect(repository.getWorkspaceProjection).toHaveBeenCalledTimes(1); + }); +}); +``` + +- [ ] **Step 3: Run to verify failure** + +Run: `cd extralit-frontend && npx vitest run v2/domain/usecases/get-workspace-projection-use-case.test.ts` +Expected: FAIL — module not found. + +- [ ] **Step 4: Implement the use-case** + +Create `v2/domain/usecases/get-workspace-projection-use-case.ts`: + +```ts +import { WorkspaceProjection } from "../entities/projection/WorkspaceProjection"; +import { ProjectionRepository } from "~/v2/infrastructure/repositories/ProjectionRepository"; +import { type useExtractions } from "~/v2/infrastructure/storage/ExtractionsStorage"; + +// Server caps limit at 100; the client loads all pages into ONE Perspective table +// (spec §3.3 data path — Arrow IPC streaming is a recorded follow-up, §5). +export const PROJECTION_PAGE_SIZE = 100; + +export class GetWorkspaceProjectionUseCase { + constructor( + private readonly projectionRepository: ProjectionRepository, + // ts-injecty resolves the hook by calling it; the injected value is the store object. + private readonly extractionsStorage: ReturnType + ) {} + + async execute(workspaceId: string): Promise { + const first = await this.projectionRepository.getWorkspaceProjection( + workspaceId, + 0, + PROJECTION_PAGE_SIZE + ); + const rows = [...first.rows]; + for (let offset = PROJECTION_PAGE_SIZE; offset < first.totalReferences; offset += PROJECTION_PAGE_SIZE) { + const page = await this.projectionRepository.getWorkspaceProjection( + workspaceId, + offset, + PROJECTION_PAGE_SIZE + ); + rows.push(...page.rows); + } + const projection = new WorkspaceProjection(first.columns, rows, first.totalReferences); + this.extractionsStorage.saveProjection(projection); + return projection; + } +} +``` + +- [ ] **Step 5: Run to verify pass** + +Run: `cd extralit-frontend && npx vitest run v2/domain/usecases/get-workspace-projection-use-case.test.ts` +Expected: ALL PASS. + +- [ ] **Step 6: Register in DI** + +In `v2/di/di.ts` add the imports: + +```ts +import { GetWorkspaceProjectionUseCase } from "~/v2/domain/usecases/get-workspace-projection-use-case"; +import { useExtractions } from "~/v2/infrastructure/storage/ExtractionsStorage"; +``` + +and, in the `dependencies` array right after the existing `register(ProjectionRepository)…` line: + +```ts + register(GetWorkspaceProjectionUseCase).withDependencies(ProjectionRepository, useExtractions).build(), +``` + +- [ ] **Step 7: Verify the suite still passes and commit** + +Run: `cd extralit-frontend && npm run test` +Expected: ALL PASS. + +```bash +git add extralit-frontend/v2/infrastructure/storage/ExtractionsStorage.ts extralit-frontend/v2/domain/usecases/get-workspace-projection-use-case.ts extralit-frontend/v2/domain/usecases/get-workspace-projection-use-case.test.ts extralit-frontend/v2/di/di.ts +git commit -m "feat(v2-ui): workspace-projection use-case, extractions storage, DI wiring" +``` + +--- + +### Phase 1 exit gate — verify and open PR 1 + +- [ ] **Step 1: Server suites** + +```bash +docker run -d --rm --name tmp-redis -p 6379:6379 redis:7 # if not already running +cd extralit-server && uv run pytest tests/unit/validators/v2 tests/integration/contexts/v2 tests/integration/api/v2 -q +``` +Expected: ALL PASS. + +- [ ] **Step 2: Contract drift gate** + +```bash +cd extralit-frontend && npm run gen:api && git diff --exit-code v2/infrastructure/api +``` +Expected: exit 0 — the snapshot committed in Task 5 matches the server. + +- [ ] **Step 3: Frontend gates** + +```bash +cd extralit-frontend && npm run test && npx nuxi typecheck && npm run lint +``` +Expected: all green. (No `npm run build` needed yet — no new build-affecting config in this phase.) + +- [ ] **Step 4: Scope audit** + +Confirm Phase 1 is purely additive: no `package.json` dependency changes, no `nuxt.config.ts`/`vitest.config.ts` changes, no page/component/e2e changes, no deletions. Existing v2 e2e specs (`review-loop`, `draft-lifecycle`, `slashed-reference`, `auth-smoke`, `search-roundtrip`) are untouched and still the active gate. + +- [ ] **Step 5: Open PR 1** + +Use superpowers:finishing-a-development-branch — PR from `feat/v2-ui-extraction-table` to `develop`: "feat(v2): workspace projection endpoint + extraction-table data layer". Then branch Phase 2: + +```bash +git checkout -b feat/v2-ui-extraction-grid +``` + +--- + +## Phase 2 — Perspective grid + page + review-page retirement (PR 2, stacked on Phase 1) + +Tasks 8–13, on `feat/v2-ui-extraction-grid`. This phase carries the integration risk this split exists to isolate: the `@perspective-dev/*` 4.5.2 WASM boot, Vite/esbuild config, the `` custom element under Vue, and the regular-table styling/click hooks — plus the destructive change (review-page deletion) and its replacement e2e gate. Task 12's e2e spec is the acceptance gate for the whole Perspective wiring; if 4.5.2 fights back, nothing in the merged Phase 1 needs to move. + +### Task 8: Perspective dependencies, Nuxt/Vite config, WASM bootstrap + test stub (frontend) + +**Files:** +- Modify: `extralit-frontend/package.json` (+ lockfile) +- Modify: `extralit-frontend/nuxt.config.ts` +- Modify: `extralit-frontend/vitest.config.ts` +- Create: `extralit-frontend/components/v2/extractions/perspective-bootstrap.ts` +- Create: `extralit-frontend/__mocks__/perspective-bootstrap.js` + +**Interfaces:** +- Produces: `initPerspective(): Promise` — module-level singleton; resolves once WASM is initialized and returns the `@perspective-dev/client` default export (so callers do `const perspective = await initPerspective(); const client = await perspective.worker();`). Task 9 imports it via the alias-stable specifier `~/components/v2/extractions/perspective-bootstrap`. +- The vitest stub replaces the whole bootstrap module, so unit tests never touch WASM/custom elements (mirrors the `tabulator-tables` mock precedent). + +- [ ] **Step 1: Install pinned packages** + +```bash +cd extralit-frontend && npm install --save-exact @perspective-dev/client@4.5.2 @perspective-dev/server@4.5.2 @perspective-dev/viewer@4.5.2 @perspective-dev/viewer-datagrid@4.5.2 +``` + +- [ ] **Step 2: Verify the WASM asset paths** (verified against the 4.5.2 tarballs on 2026-07-20, but confirm locally) + +```bash +ls node_modules/@perspective-dev/server/dist/wasm/perspective-server.wasm node_modules/@perspective-dev/viewer/dist/wasm/perspective-viewer.wasm +``` +Expected: both files listed. If a path differs, use the actual path in Step 4's `?url` imports. + +- [ ] **Step 3: Nuxt/Vite config** + +In `nuxt.config.ts` add a top-level `vue` key (there is none today) and extend the existing `vite` block (currently ~lines 88–107; keep the existing `plugins`, `server`, `css`, and `optimizeDeps.include` entries): + +```ts + vue: { + compilerOptions: { + // is a web Custom Element, not a Vue component (spec §3.3). + isCustomElement: (tag: string) => tag.startsWith("perspective-"), + }, + }, +``` + +```ts + vite: { + // …existing entries stay… + build: { + target: "esnext", // Perspective 4.x ESM/WASM requirement + }, + optimizeDeps: { + include: [/* …existing entries stay… */], + // WASM ESM packages break under esbuild pre-bundling; load them as-is. + exclude: [ + "@perspective-dev/client", + "@perspective-dev/server", + "@perspective-dev/viewer", + "@perspective-dev/viewer-datagrid", + ], + esbuildOptions: { + target: "esnext", + }, + }, + }, +``` + +- [ ] **Step 4: Create the bootstrap module** + +Create `components/v2/extractions/perspective-bootstrap.ts`: + +```ts +import perspective from "@perspective-dev/client"; +import perspective_viewer from "@perspective-dev/viewer"; +import "@perspective-dev/viewer-datagrid"; +import SERVER_WASM from "@perspective-dev/server/dist/wasm/perspective-server.wasm?url"; +import CLIENT_WASM from "@perspective-dev/viewer/dist/wasm/perspective-viewer.wasm?url"; + +// SPA (ssr: false): this runs client-side only. Module-level guard so the WASM +// engines initialize exactly once no matter how often the page remounts (spec §3.3). +let ready: Promise | null = null; + +export const initPerspective = () => { + ready ??= Promise.all([ + perspective.init_server(fetch(SERVER_WASM)), + perspective_viewer.init_client(fetch(CLIENT_WASM)), + ]).then(() => perspective); + return ready; +}; +``` + +If TypeScript rejects the `?url` imports, add a declaration file `types/wasm-url.d.ts`: + +```ts +declare module "*.wasm?url" { + const url: string; + export default url; +} +``` + +If `init_server`/`init_client` names differ in 4.5.2, check `node_modules/@perspective-dev/client/dist/esm/perspective.d.ts` and `node_modules/@perspective-dev/viewer/dist/esm/perspective-viewer.d.ts` for the actual exported init functions and use those — the shape above matches . + +- [ ] **Step 5: Create the vitest stub and alias it** + +Create `__mocks__/perspective-bootstrap.js`: + +```js +// Perspective touches WASM + custom elements at import time; specs use this stub +// (same rationale as __mocks__/tabulator-tables.js). +export const initPerspective = async () => ({ + worker: async () => ({ + table: async (data) => ({ + __data: data, + size: async () => data.length, + delete: async () => undefined, + }), + }), +}); +``` + +In `vitest.config.ts`, add to the `resolve.alias` object ABOVE the `"~~"` entry (longer keys must win before `"~"` matches): + +```ts + // Perspective boots WASM at import; specs use the stub (see __mocks__/). + "~/components/v2/extractions/perspective-bootstrap": r("./__mocks__/perspective-bootstrap.js"), +``` + +- [ ] **Step 6: Verify config health** + +Run: `cd extralit-frontend && npm run test && npx nuxi typecheck` +Expected: suite still green; typecheck clean (nothing imports the bootstrap yet — this proves the config alone breaks nothing). + +- [ ] **Step 7: Commit** + +```bash +git add extralit-frontend/package.json extralit-frontend/package-lock.json extralit-frontend/nuxt.config.ts extralit-frontend/vitest.config.ts extralit-frontend/components/v2/extractions/perspective-bootstrap.ts extralit-frontend/__mocks__/perspective-bootstrap.js +# plus extralit-frontend/types/wasm-url.d.ts if you created it in Step 4 +git commit -m "feat(v2-ui): Perspective 4.5.2 deps, WASM bootstrap, Vite/vitest wiring" +``` + +--- + +### Task 9: `ExtractionsGrid.vue` — viewer wrapper with banding + click plumbing (frontend) + +**Files:** +- Create: `extralit-frontend/components/v2/extractions/ExtractionsGrid.vue` +- Test: `extralit-frontend/components/v2/extractions/ExtractionsGrid.test.ts` + +**Interfaces:** +- Consumes: `initPerspective` (Task 8), `toPerspectiveData`/`cellAt`/`bandParity` (Task 6), `WorkspaceProjection` (Task 5). +- Produces: `` (auto-imported by name — `components` config uses `pathPrefix: false`). Emits `cell-click` with `{ cell: ProjectionGridCell; reference: string; schemaId: string; columnName: string }`. Task 10's page listens to this. +- Lazy chunking (spec §3.3): Nuxt code-splits per page and auto-imported components bundle into the chunks that use them — since only `pages/extractions/index.vue` uses this component, the Perspective JS/WASM cost is paid only on `/extractions`. Verify in Task 13's `npm run build` output: the perspective modules must NOT appear in the entry chunk. +- Note: the custom-element/regular-table interactions cannot be exercised under happy-dom; the unit test covers mount → bootstrap → table creation → viewer load. The real rendering gate is Task 12's e2e spec. + +- [ ] **Step 1: Write the failing component test** + +Create `components/v2/extractions/ExtractionsGrid.test.ts`: + +```ts +import { flushPromises, mount } from "@vue/test-utils"; +import { describe, expect, it, vi } from "vitest"; +import ExtractionsGrid from "./ExtractionsGrid.vue"; +import { WorkspaceProjection } from "~/v2/domain/entities/projection/WorkspaceProjection"; + +const tableSpy = vi.fn(async (data: unknown) => ({ __data: data, delete: async () => undefined })); +const initSpy = vi.fn(async () => ({ worker: async () => ({ table: tableSpy }) })); + +vi.mock("~/components/v2/extractions/perspective-bootstrap", () => ({ + initPerspective: (...args: unknown[]) => initSpy(...args), +})); + +const PROJECTION = new WorkspaceProjection( + [{ name: "Design.type", schemaId: "s-1", schemaName: "Design", questionName: "type", subColumn: null, dtype: "text" }], + [ + { + reference: "10.1/a", + rowIndex: 0, + cells: { + "Design.type": { value: "RCT", source: "response", recordId: "r-1", agent: null, score: null }, + }, + }, + ], + 1 +); + +describe("ExtractionsGrid", () => { + it("boots perspective once and loads the flat projection rows into a table", async () => { + mount(ExtractionsGrid, { + props: { projection: PROJECTION }, + global: { + config: { compilerOptions: { isCustomElement: (tag: string) => tag.startsWith("perspective-") } }, + }, + }); + await flushPromises(); + + expect(initSpy).toHaveBeenCalledTimes(1); + expect(tableSpy).toHaveBeenCalledWith([{ reference: "10.1/a", "Design.type": "RCT" }]); + }); +}); +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cd extralit-frontend && npx vitest run components/v2/extractions/ExtractionsGrid.test.ts` +Expected: FAIL — component file not found. + +- [ ] **Step 3: Implement the component** + +Create `components/v2/extractions/ExtractionsGrid.vue`: + +```vue + + + + + +``` + +Note: if `restore({ plugin: "Datagrid" })` names the plugin differently in 4.5.2, the accepted values are listed by `viewer.restore` typings in `node_modules/@perspective-dev/viewer/dist/esm/perspective-viewer.d.ts` — the datagrid plugin registers as `"Datagrid"`. The `:deep()` selectors only apply if the datagrid renders in light DOM (it is slotted into the viewer); if banding doesn't show in Task 12's e2e run, move the two rules to an unscoped ` +``` + +(`useEnsureWorkspaces` and `useV2Breadcrumbs` are Nuxt auto-imported composables — match how `pages/schemas/index.vue` references them; if that file imports them explicitly, do the same here.) + +- [ ] **Step 7: Full check + smoke** + +Run: `cd extralit-frontend && npm run test && npx nuxi typecheck && npm run lint` +Expected: all green. + +Optional smoke (backend on :6900): `npm run dev` → open `http://localhost:3000/extractions` → page renders (empty state or grid). + +- [ ] **Step 8: Commit** + +```bash +git add extralit-frontend/pages/extractions extralit-frontend/composables/useV2Breadcrumbs.ts extralit-frontend/translation +git commit -m "feat(v2-ui): /extractions page with Perspective grid and workspace override" +``` + +--- + +### Task 11: Delete the reference-review page (frontend) + +**Files:** +- Create: `extralit-frontend/v2/domain/entities/review/ReviewCell.ts` (type extraction so kept files survive) +- Modify: `components/v2/review/ReviewCellInput.vue`, `components/v2/review/ReviewCellInput.test.ts`, `components/v2/review/ReviewProvenance.vue`, `v2/domain/entities/review/widget-adapters.ts`, `v2/domain/entities/review/widget-adapters.test.ts`, `v2/di/di.ts`, `components/v2/schemas/V2RecordsTable.vue` (+ its `.test.ts` if it asserts the link) +- Delete: `pages/references/[...reference].vue`, `pages/references/useReferenceReviewViewModel.ts`, `pages/references/useReferenceReviewViewModel.test.ts`, `components/v2/review/ProjectionReviewForm.vue` (+ co-located `.test.ts`), `components/v2/review/ReviewRecordCard.vue` (+ co-located `.test.ts` if present), `v2/domain/entities/review/ReferenceReview.ts`, `v2/domain/usecases/get-reference-review-use-case.ts`, `v2/domain/usecases/get-reference-review-use-case.test.ts`, `v2/infrastructure/storage/ReferenceReviewsStorage.ts`, `e2e/v2/review-loop.spec.ts`, `e2e/v2/draft-lifecycle.spec.ts`, `e2e/v2/slashed-reference.spec.ts` + +**Interfaces:** +- Produces: `v2/domain/entities/review/ReviewCell.ts` exporting `Provenance` and `ReviewCell` — the ONLY review types the kept drawer-reuse files need (`ReviewRecord`/`ReferenceReview`/`ContextField`/`OrphanedValue` die with the page). Kept and still registered in DI: `SubmitReferenceReviewUseCase`, `SaveReviewDraftUseCase`, `DiscardReviewUseCase`, `ProjectionRepository`. + +- [ ] **Step 1: Extract the kept types** + +Create `v2/domain/entities/review/ReviewCell.ts` (moved verbatim from `ReferenceReview.ts`): + +```ts +import { Question } from "../question/Question"; + +export interface Provenance { + agent: string | null; + score: number | null; + suggestedValue: unknown; +} + +export class ReviewCell { + constructor( + public readonly question: Question, + public readonly value: unknown, + public readonly source: "response" | "suggestion" | null, + public readonly provenance: Provenance | null, + // The question binds a column absent from this record's pinned version cache (§17.3). + public readonly notApplicable: boolean + ) {} +} +``` + +- [ ] **Step 2: Repoint the kept importers** + +- `components/v2/review/ReviewCellInput.vue` line ~55: `import { type ReviewCell } from "~/v2/domain/entities/review/ReviewCell";` +- `components/v2/review/ReviewCellInput.test.ts` line ~5: same repoint (import `ReviewCell` — and `Provenance` if it uses it — from `~/v2/domain/entities/review/ReviewCell`). +- `components/v2/review/ReviewProvenance.vue` line ~13: `import { type Provenance } from "~/v2/domain/entities/review/ReviewCell";` +- `v2/domain/entities/review/widget-adapters.ts` line ~2: `import { type ReviewCell } from "./ReviewCell";` +- `v2/domain/entities/review/widget-adapters.test.ts` line ~3: same repoint. + +- [ ] **Step 3: Delete the superseded files** + +```bash +cd extralit-frontend +git rm pages/references/[...reference].vue pages/references/useReferenceReviewViewModel.ts pages/references/useReferenceReviewViewModel.test.ts +git rm components/v2/review/ProjectionReviewForm.vue components/v2/review/ReviewRecordCard.vue +git rm --ignore-unmatch components/v2/review/ProjectionReviewForm.test.ts components/v2/review/ReviewRecordCard.test.ts +git rm v2/domain/entities/review/ReferenceReview.ts +git rm v2/domain/usecases/get-reference-review-use-case.ts v2/domain/usecases/get-reference-review-use-case.test.ts +git rm v2/infrastructure/storage/ReferenceReviewsStorage.ts +git rm e2e/v2/review-loop.spec.ts e2e/v2/draft-lifecycle.spec.ts e2e/v2/slashed-reference.spec.ts +``` + +- [ ] **Step 4: Clean DI** + +In `v2/di/di.ts` remove the imports of `GetReferenceReviewUseCase` and `useReferenceReviews`, and remove the whole `register(GetReferenceReviewUseCase).withDependencies(…).build(),` entry. KEEP `SubmitReferenceReviewUseCase`, `SaveReviewDraftUseCase`, `DiscardReviewUseCase`, and `ProjectionRepository` registrations (they power the future embedded drawer). + +- [ ] **Step 5: Retarget the records-table reference cell** + +In `components/v2/schemas/V2RecordsTable.vue` (lines ~13–19) replace the `NuxtLink` with plain text, keeping `data-reference` (e2e hooks use it): + +```vue + + {{ record.reference }} + +``` + +If `components/v2/schemas/V2RecordsTable.test.ts` asserts the link/href, update those assertions to the span + `data-reference` attribute. + +- [ ] **Step 6: Sweep for dead references** + +```bash +cd extralit-frontend && grep -rn "references/" pages components v2 e2e composables --include="*.ts" --include="*.vue" | grep -v "projection/references" | grep -v node_modules +grep -rn "ReferenceReview\|useReferenceReviews\|ProjectionReviewForm\|ReviewRecordCard" pages components v2 e2e --include="*.ts" --include="*.vue" +``` +Expected: no hits outside `ReviewCell.ts` comments/kept files' repointed imports. Fix any stragglers the same way (repoint or delete). + +- [ ] **Step 7: Verify everything still passes** + +Run: `cd extralit-frontend && npm run test && npx nuxi typecheck && npm run lint` +Expected: all green; test count drops by the deleted specs only. + +- [ ] **Step 8: Commit** + +```bash +git add -A extralit-frontend +git commit -m "refactor(v2-ui): delete reference-review page — review derives from projection (spec §3.5)" +``` + +--- + +### Task 12: e2e — seed extension + `extractions-grid.spec.ts` + +**Files:** +- Modify: `extralit-frontend/e2e/v2/seed/seed_v2_e2e.py` +- Modify: `extralit-frontend/e2e/v2/fixtures.ts` (extend `SeedOutput`) +- Create: `extralit-frontend/e2e/v2/extractions-grid.spec.ts` + +**Interfaces:** +- Consumes: existing `test`/`expect`/`signIn`/`loadSeed` from `e2e/v2/fixtures.ts`; the `/extractions?workspace_id=` override (Task 10). +- Produces: seed additionally creates (a) a submitted response for the `label` question (value `"control"`) plus a competing suggestion (`"intervention"`) so the grid proves response-beats-suggestion, and (b) a second schema `e2e_v2_empty` with one question and ZERO records (coverage map). `SeedOutput` gains `emptySchemaName: string`. + +- [ ] **Step 1: Extend the seed script** + +In `e2e/v2/seed/seed_v2_e2e.py`: + +Top-of-file constants (next to `SCHEMA_NAME`): + +```python +EMPTY_SCHEMA_NAME = "e2e_v2_empty" + +EMPTY_BODY = pa.DataFrameSchema( + columns={"notes": pa.Column(pa.String, nullable=True)} +).to_json() +``` + +Extend the schema-recreate loop (currently deletes only `SCHEMA_NAME`): + +```python + for schema_item in schemas: + if schema_item["name"] in (SCHEMA_NAME, EMPTY_SCHEMA_NAME): + client.delete(f"/api/v2/schemas/{schema_item['id']}").raise_for_status() +``` + +After the existing suggestion `client.put(...suggestions...)` block, add: + +```python + # Response-beats-suggestion on `label`: competing suggestion + submitted response. + client.put( + f"/api/v2/records/{record['id']}/suggestions", + json={ + "question_id": questions["label"]["id"], + "value": "intervention", + "score": 0.55, + "agent": "e2e-seeder", + }, + ).raise_for_status() + client.put( + f"/api/v2/records/{record['id']}/responses", + json={"values": {"label": {"value": "control"}}, "status": "submitted"}, + ).raise_for_status() + + # Second schema with zero records: the grid doubles as a coverage map (§3.1). + empty_schema = ( + client.post( + "/api/v2/schemas", + json={"name": EMPTY_SCHEMA_NAME, "workspace_id": workspace["id"]}, + ) + .raise_for_status() + .json() + ) + client.post( + f"/api/v2/schemas/{empty_schema['id']}/versions", json={"body": EMPTY_BODY} + ).raise_for_status() + client.post( + f"/api/v2/schemas/{empty_schema['id']}/questions", + json={ + "name": "notes", + "title": "Notes", + "type": "text", + "columns": ["notes"], + "settings": {}, + "required": False, + }, + ).raise_for_status() +``` + +And extend the output dict: + +```python + "emptySchemaName": EMPTY_SCHEMA_NAME, +``` + +In `e2e/v2/fixtures.ts`, add `emptySchemaName: string;` to the `SeedOutput` type. + +- [ ] **Step 2: Write the spec** + +Create `e2e/v2/extractions-grid.spec.ts`: + +```ts +import { expect, loadSeed, signIn, test } from "./fixtures"; + +// Replacement gate for the deleted review-loop specs (spec §3.5 accepted risk): +// seed → grid renders coalesced values → coverage-gap columns present. +test("extraction table renders coalesced values and coverage gaps", async ({ page }) => { + const seed = loadSeed(); + await signIn(page); + + const projectionRequest = page.waitForResponse( + (r) => + r.url().includes("/api/v2/projection") && + !r.url().includes("/references/") && + r.request().method() === "GET" + ); + await page.goto(`/extractions?workspace_id=${seed.workspaceId}`); + expect((await projectionRequest).status()).toBe(200); + + const viewer = page.locator("perspective-viewer"); + await expect(viewer).toBeVisible(); + + // Column manifest: both schemas appear — including the record-less one (coverage map). + await expect(page.getByText(`${seed.schemaName}.size`)).toBeVisible(); + await expect(page.getByText(`${seed.emptySchemaName}.notes`)).toBeVisible(); + + // Rows + coalesced values: suggestion-sourced `size`, response-beats-suggestion `label`. + await expect(page.getByText(seed.reference).first()).toBeVisible(); + await expect(page.getByText("120").first()).toBeVisible(); + await expect(page.getByText("control").first()).toBeVisible(); + await expect(page.getByText("intervention")).toHaveCount(0); +}); +``` + +- [ ] **Step 3: Run against the live stack** + +Runbook (Orin host — see `extralit-frontend/CLAUDE.md` "v2 e2e suite" and the server-env notes): + +```bash +docker run -d --rm --name tmp-redis -p 6379:6379 redis:7 # if not already up +cd extralit-server && MINIO_HOST=localhost ELASTICSEARCH_HOST=localhost REDIS_URL=redis://localhost:6379 uv run python -m extralit_server server-dev & # or however the stack is normally brought up +cd extralit-frontend +npm run e2e:v2:seed +npm run dev -- --host & +npm run e2e:v2 +``` + +Expected: `extractions-grid.spec.ts` PASSES, and the remaining v2 specs (`auth-smoke`, `search-roundtrip`) stay green (the seed's new response/suggestion must not break `search-roundtrip` — if it asserts on the `label` record fields, re-run `npm run e2e:v2:seed` and check its assertions). + +If Perspective fails to boot in the real browser (WASM path / init API mismatch), fix in Task 8's bootstrap or Task 9's restore config — the exact d.ts files to consult are noted in those tasks. This spec is the acceptance gate for the whole Perspective wiring. + +- [ ] **Step 4: Commit** + +```bash +git add extralit-frontend/e2e/v2/seed/seed_v2_e2e.py extralit-frontend/e2e/v2/fixtures.ts extralit-frontend/e2e/v2/extractions-grid.spec.ts +git commit -m "test(v2-ui): extractions-grid e2e gate with coverage-map seed" +``` + +--- + +### Task 13: Phase 2 final verification — full suites + drift gate + +**Files:** none new (fix-ups only if something fails). Re-runs the Phase 1 gates too, since Phase 2 sits on top of them. + +- [ ] **Step 1: Server suite** + +```bash +docker run -d --rm --name tmp-redis -p 6379:6379 redis:7 # if not already running +cd extralit-server && uv run pytest tests/unit/validators/v2 tests/integration/contexts/v2 tests/integration/api/v2 -q +``` +Expected: ALL PASS. + +- [ ] **Step 2: Contract drift gate** + +```bash +cd extralit-frontend && npm run gen:api && git diff --exit-code v2/infrastructure/api +``` +Expected: exit 0 (no drift — the committed snapshot matches the server). + +- [ ] **Step 3: Frontend gates** + +```bash +cd extralit-frontend && npm run test && npx nuxi typecheck && npm run lint && npm run build +``` +Expected: all green; `npm run build` proves the Perspective WASM `?url` imports and `esnext` target survive a production build. + +- [ ] **Step 4: Scope audit against the spec** + +Confirm: no nav/`index.vue` wiring for `/extractions`; no sort/filter UI; no Arrow IPC; no annotation-mode changes; `ANNOTATION_CELL_LINKS_ENABLED === false`; kept review files intact (spec §3.5 keep-list); `V2TableEditor.vue` untouched (still emits the dict form). + +- [ ] **Step 5: Commit any fix-ups** + +```bash +git status # should be clean; commit stragglers with a descriptive message if not +``` + +Then hand off with superpowers:finishing-a-development-branch — PR 2 from `feat/v2-ui-extraction-grid` to `develop` ("feat(v2-ui): /extractions Perspective grid, review-page retirement"). If PR 1 has merged, rebase onto `develop` first so PR 2's diff shows only Phase 2 commits; if not, open it stacked on `feat/v2-ui-extraction-table` and retarget after PR 1 lands. From f133812f909f28c44b4fdc5361451cbcb4d28a5e Mon Sep 17 00:00:00 2001 From: JonnyTran Date: Mon, 20 Jul 2026 23:41:33 -0700 Subject: [PATCH 04/21] docs(plan): update extraction table plan with DuckDB integration and architectural details - Enhanced the architecture section to detail the use of DuckDB for in-memory denormalization, replacing previous methods. - Clarified the tech stack to include DuckDB as a denormalization engine. - Updated Phase 1 description to specify the addition of DuckDB as a new dependency. - Added new integration tests for the workspace projection functionality. --- .../plans/2026-07-20-extraction-table.md | 299 +++++++++++------- 1 file changed, 189 insertions(+), 110 deletions(-) diff --git a/docs/superpowers/plans/2026-07-20-extraction-table.md b/docs/superpowers/plans/2026-07-20-extraction-table.md index fb86d3c15..2c67cb81d 100644 --- a/docs/superpowers/plans/2026-07-20-extraction-table.md +++ b/docs/superpowers/plans/2026-07-20-extraction-table.md @@ -5,12 +5,12 @@ **Goal:** Ship a workspace-level denormalized extraction table — new `GET /api/v2/projection` endpoint with enriched provenance cells, a `/extractions` page rendered by Perspective 4.x, an additive `list[dict]` table-value contract, and deletion of the superseded reference-review page. **Delivery: two phases, two PRs.** -- **Phase 1 (Tasks 1–7, PR 1):** everything additive and low-risk — server contract (validator extension, enriched cells, workspace endpoint) plus the frontend data layer (gen:api, domain types, repository, grid adapter, use-case, storage, DI). No UI, no deletions, no new dependencies. Mergeable to `develop` on its own; SDK #231 and any other consumer can build against the endpoint immediately. +- **Phase 1 (Tasks 1–7, PR 1):** everything additive and low-risk — server contract (validator extension, enriched cells, workspace endpoint) plus the frontend data layer (gen:api, domain types, repository, grid adapter, use-case, storage, DI). No UI, no deletions, no new frontend dependencies (the server gains `duckdb`). Mergeable to `develop` on its own; SDK #231 and any other consumer can build against the endpoint immediately. - **Phase 2 (Tasks 8–13, PR 2, stacked on Phase 1):** the integration-risk half — Perspective deps/WASM/Vite wiring, the custom-element grid wrapper, the `/extractions` page, deletion of the reference-review page, and the e2e gate. If Perspective 4.5.2 fights back (WASM boot, init API drift, custom-element quirks), Phase 1 is already merged and unaffected. -**Architecture:** The server performs the full denormalization (coalesce `submitted response → suggestion → empty`, table fan-out, independent stacking, scalar repetition) in `contexts/v2/projection.py` using batched queries; the client only pages, aggregates, and renders. Frontend follows the existing v2 DDD chain: `ProjectionRepository` → `GetWorkspaceProjectionUseCase` → Pinia storage → `useExtractionsViewModel` → `pages/extractions/index.vue`, with a Perspective web-component grid wrapped in `ExtractionsGrid.vue`. +**Architecture:** The server performs the full denormalization in `contexts/v2/projection.py`: batched Postgres queries (via the existing `AsyncSession`) fetch raw slices, and an **in-memory DuckDB** connection runs ONE SQL statement that does everything semantic — effective-record dedup (window functions), `submitted response ?? suggestion` coalesce, table fan-out (`json_each`), the independent-stacking row spine, and scalar repetition. Python only registers inputs and regroups the long-format result into Pydantic models, so the transform stays declarative over arbitrary schema-defined JSON and pre-builds the spec §5 Arrow path (`.fetchall()` → `.arrow()` later streams Arrow IPC straight into Perspective). The client only pages, aggregates, and renders. Frontend follows the existing v2 DDD chain: `ProjectionRepository` → `GetWorkspaceProjectionUseCase` → Pinia storage → `useExtractionsViewModel` → `pages/extractions/index.vue`, with a Perspective web-component grid wrapped in `ExtractionsGrid.vue`. -**Tech Stack:** FastAPI + SQLAlchemy async (extralit-server), Vue 3 / Nuxt 4 / Pinia / ts-injecty (extralit-frontend), Perspective `@perspective-dev/*` 4.5.2 (WASM), openapi-typescript contract gate, pytest / vitest / Playwright. +**Tech Stack:** FastAPI + SQLAlchemy async + DuckDB (in-process denormalization engine) on extralit-server, Vue 3 / Nuxt 4 / Pinia / ts-injecty (extralit-frontend), Perspective `@perspective-dev/*` 4.5.2 (WASM), openapi-typescript contract gate, pytest / vitest / Playwright. **Spec:** `docs/superpowers/specs/2026-07-20-extraction-table-design.md` (this plan implements §3–§4 within the §6 scope boundary; §5 items are explicitly NOT built). @@ -31,7 +31,7 @@ **extralit-server (modify):** - `src/extralit_server/validators/v2/values.py` — `_validate_table` accepts `list[dict]` additively (Task 1) - `src/extralit_server/api/schemas/v2/projection.py` — enrich `ProjectionCell`; add `WorkspaceProjection*` models (Tasks 2, 3) -- `src/extralit_server/contexts/v2/projection.py` — enrich `build_reference_view`; add `build_workspace_view` + pure denormalization helpers (Tasks 2, 3) +- `src/extralit_server/contexts/v2/projection.py` — enrich `build_reference_view`; add `build_workspace_view` + the DuckDB denormalization statement (Tasks 2, 3) - `src/extralit_server/api/v2/projection.py` — add `GET /projection` route (Task 4) - `tests/unit/validators/v2/test_values.py`, `tests/integration/contexts/v2/test_projection.py`, `tests/integration/api/v2/test_projection.py` — extend; new `tests/integration/contexts/v2/test_workspace_projection.py` @@ -246,13 +246,15 @@ git commit -m "feat(server): enrich ProjectionCell with record_id/agent/score pr **Files:** - Modify: `extralit-server/src/extralit_server/api/schemas/v2/projection.py` (add 4 models) -- Modify: `extralit-server/src/extralit_server/contexts/v2/projection.py` (add `build_workspace_view` + helpers) +- Modify: `extralit-server/src/extralit_server/contexts/v2/projection.py` (add `build_workspace_view` + the DuckDB denormalization) +- Modify: `extralit-server/pyproject.toml` + `uv.lock` (via `uv add duckdb`) - Create: `extralit-server/tests/integration/contexts/v2/test_workspace_projection.py` **Interfaces:** - Consumes: `Schema`, `V2Question` (`.columns` binding, `.type`), `V2Record` (`.reference`), `V2Suggestion`, `V2Response` ORM models; `ResponseStatus`, `QuestionType` enums. - Produces: `async def build_workspace_view(db: AsyncSession, *, workspace_id: UUID, offset: int, limit: int) -> WorkspaceProjection` — Task 4's route calls exactly this. Pydantic models `WorkspaceProjectionColumn`, `WorkspaceProjectionCell`, `WorkspaceProjectionRow`, `WorkspaceProjection` (names Task 4 and gen:api rely on). - Semantics locked here: `limit`/`offset` count **references** (not fan-out rows); cell coalesce = **latest submitted response by any user** (by `updated_at`) `??` suggestion `??` omitted; one **effective record** per (reference, schema) = latest `inserted_at`; table fan-out with **independent stacking** (row count = max fan-out, min 1) and **scalar repetition**; absent cells omitted from `cells`. +- Engine: Postgres (via the existing `AsyncSession`) serves only batched raw slices; an **in-memory DuckDB** connection runs ONE SQL statement implementing all of the semantics above (window-function dedup, `json_each` fan-out, spine + repetition) over the schema-defined JSON. Do NOT `ATTACH` Postgres from DuckDB — integration tests run inside a rolled-back transaction that a second connection can't see, and it would add a second credential/pool path. The tests and API contract are engine-agnostic: they exercise `build_workspace_view` only. - [ ] **Step 1: Add the Pydantic models** @@ -507,11 +509,23 @@ Note: if `V2ResponseFactory`/`V2QuestionFactory` kwargs differ (check `tests/fac Run: `cd extralit-server && uv run pytest tests/integration/contexts/v2/test_workspace_projection.py -v` Expected: FAIL — `module … has no attribute 'build_workspace_view'`. -- [ ] **Step 4: Implement** +- [ ] **Step 4: Add the DuckDB dependency** + +```bash +cd extralit-server && uv add duckdb +``` + +DuckDB (in-process) is the denormalization engine: the transform stays declarative SQL over the schema-defined JSON instead of nested Python loops, and swapping `.fetchall()` for `.arrow()` later yields the spec §5 Arrow-IPC streaming path for free. Its bundled JSON extension autoloads — no `INSTALL`/`LOAD` calls needed. `anyio` (used below for the thread offload) already ships with FastAPI/Starlette. + +- [ ] **Step 5: Implement** In `src/extralit_server/contexts/v2/projection.py`, extend the imports (keep existing ones): ```python +import json + +import duckdb +from anyio import to_thread from sqlalchemy import func, select from extralit_server.api.schemas.v2.projection import ( @@ -563,81 +577,119 @@ def _build_columns( return columns -def _resolve(record, question, suggestions, responses): - """Coalesce (spec §3.1): latest submitted response (any user) ?? suggestion ?? None. - Returns (value, source, agent, score) or None. Drafts never reach `responses`.""" - wrapped = responses.get(record.id, {}).get(question.name) - if wrapped is not None: - return (wrapped.get("value"), "response", None, None) - suggestion = suggestions.get((record.id, question.id)) - if suggestion is not None: - return (suggestion.value, "suggestion", suggestion.agent, suggestion.score) - return None - - -def _table_rows(value) -> list[dict]: - # §3.4 normalization: bare dict is the 1-row case; list[dict] is N rows. - if isinstance(value, list): - return [row for row in value if isinstance(row, dict)] - if isinstance(value, dict): - return [value] - return [] - +# One SQL statement performs the entire denormalization (spec §3.1/§3.4): effective- +# record dedup, response ?? suggestion coalesce, table fan-out, the independent-stacking +# row spine, and scalar repetition. Long format out: one row per (reference, row_idx, +# column) — plus a bare spine row when a reference has no resolved cells yet. +_DENORMALIZE_SQL = """ +WITH effective_records AS ( -- one effective record per (reference, schema): latest inserted_at + SELECT record_id, schema_id, reference + FROM records + QUALIFY row_number() OVER (PARTITION BY reference, schema_id ORDER BY inserted_at DESC) = 1 +), +latest_responses AS ( -- latest submitted response per record, ANY user (user-agnostic view) + SELECT record_id, values_json + FROM responses + QUALIFY row_number() OVER (PARTITION BY record_id ORDER BY updated_at DESC) = 1 +), +response_cells AS ( -- explode the {question_name: {"value": ...}} envelope + SELECT lr.record_id, je.key AS question_name, json_extract(je.value, '$.value') AS value + FROM latest_responses lr, json_each(lr.values_json) je +), +resolved AS ( -- coalesce: response ?? suggestion; (record, question) with neither drops out + SELECT er.reference, er.record_id, q.question_id, q.schema_name, q.question_name, q.qtype, + COALESCE(rc.value, s.value_json) AS value, + CASE WHEN rc.value IS NOT NULL THEN 'response' ELSE 'suggestion' END AS source, + CASE WHEN rc.value IS NULL THEN s.agent END AS agent, + CASE WHEN rc.value IS NULL THEN s.score_json END AS score + FROM effective_records er + JOIN questions q ON q.schema_id = er.schema_id + LEFT JOIN response_cells rc + ON rc.record_id = er.record_id AND rc.question_name = q.question_name + LEFT JOIN suggestions s + ON s.record_id = er.record_id AND s.question_id = q.question_id + WHERE COALESCE(rc.value, s.value_json) IS NOT NULL +), +table_rows AS ( -- §3.4 in SQL: bare dict = the 1-row case; arrays fan out with 0-based indices + SELECT r.reference, r.record_id, r.question_id, r.schema_name, r.question_name, + r.source, r.agent, r.score, + CAST(je.key AS BIGINT) AS row_idx, je.value AS row_json + FROM resolved r, + json_each(CASE WHEN json_type(r.value) = 'ARRAY' THEN r.value + ELSE json_array(r.value) END) je + WHERE r.qtype = 'table' AND json_type(je.value) = 'OBJECT' +), +table_cells AS ( -- fan each row dict out over the question's bound sub-columns + SELECT tr.reference, tr.row_idx, + tr.schema_name || '.' || tr.question_name || '.' || qc.sub_column AS column_name, + json_extract(tr.row_json, '$."' || qc.sub_column || '"') AS value, + tr.source, tr.record_id, tr.agent, tr.score + FROM table_rows tr + JOIN question_columns qc ON qc.question_id = tr.question_id + WHERE COALESCE(json_type(json_extract(tr.row_json, '$."' || qc.sub_column || '"')), 'NULL') <> 'NULL' +), +scalar_cells AS ( + SELECT reference, schema_name || '.' || question_name AS column_name, + value, source, record_id, agent, score + FROM resolved + WHERE qtype <> 'table' AND json_type(value) <> 'NULL' +), +spine AS ( -- independent stacking: rows per reference = max table fan-out, min 1 + SELECT refs.reference, unnest(generate_series(0, COALESCE(mx.mx, 0))) AS row_idx + FROM (SELECT DISTINCT reference FROM effective_records) refs + LEFT JOIN (SELECT reference, MAX(row_idx) AS mx FROM table_rows GROUP BY reference) mx + ON mx.reference = refs.reference +), +long_cells AS ( + SELECT sp.reference, sp.row_idx, sc.column_name, sc.value, + sc.source, sc.record_id, sc.agent, sc.score + FROM spine sp + JOIN scalar_cells sc ON sc.reference = sp.reference -- scalar repetition on every fan-out row + UNION ALL + SELECT reference, row_idx, column_name, value, source, record_id, agent, score + FROM table_cells +) +SELECT sp.reference, sp.row_idx, lc.column_name, CAST(lc.value AS VARCHAR) AS value_json, + lc.source, lc.record_id, lc.agent, CAST(lc.score AS VARCHAR) AS score_json +FROM spine sp +LEFT JOIN long_cells lc ON lc.reference = sp.reference AND lc.row_idx = sp.row_idx +ORDER BY sp.reference, sp.row_idx, lc.column_name +""" + + +_INPUT_DDL = { + "questions": "question_id VARCHAR, schema_id VARCHAR, schema_name VARCHAR, question_name VARCHAR, qtype VARCHAR", + "question_columns": "question_id VARCHAR, sub_column VARCHAR", + "records": "record_id VARCHAR, schema_id VARCHAR, reference VARCHAR, inserted_at TIMESTAMP", + "suggestions": "record_id VARCHAR, question_id VARCHAR, value_json JSON, agent VARCHAR, score_json JSON", + "responses": "record_id VARCHAR, values_json JSON, updated_at TIMESTAMP", +} -def _denormalize_reference( - reference: str, - *, - schemas: list[Schema], - questions_by_schema: dict, - effective: dict, - suggestions: dict, - responses: dict, -) -> list[WorkspaceProjectionRow]: - scalar_cells: dict[str, WorkspaceProjectionCell] = {} - table_values: list[tuple] = [] # (schema_name, question, rows, record, (source, agent, score)) - fan_out = 1 - for schema in schemas: - record = effective.get((reference, schema.id)) - if record is None: - continue # coverage gap: this schema's columns stay empty for the reference - for question in questions_by_schema.get(schema.id, []): - resolved = _resolve(record, question, suggestions, responses) - if resolved is None: - continue # absent cells are omitted; the client renders them as null - value, source, agent, score = resolved - if question.type == QuestionType.table: - rows = _table_rows(value) - if not rows: - continue - fan_out = max(fan_out, len(rows)) - table_values.append((schema.name, question, rows, record, (source, agent, score))) - elif value is not None: - scalar_cells[f"{schema.name}.{question.name}"] = WorkspaceProjectionCell( - value=value, source=source, record_id=record.id, agent=agent, score=score - ) - out: list[WorkspaceProjectionRow] = [] - for row_index in range(fan_out): - cells = dict(scalar_cells) # scalars repeat on every fan-out row (denormalized, §3.1) - for schema_name, question, rows, record, (source, agent, score) in table_values: - if row_index >= len(rows): - continue # independent stacking: shorter tables just end, no fabricated pairing - for sub in question.columns: - sub_value = rows[row_index].get(sub) - if sub_value is None: - continue - cells[f"{schema_name}.{question.name}.{sub}"] = WorkspaceProjectionCell( - value=sub_value, source=source, record_id=record.id, agent=agent, score=score - ) - out.append(WorkspaceProjectionRow(reference=reference, row_index=row_index, cells=cells)) - return out +def _run_denormalization(inputs: dict[str, list[tuple]]) -> list[tuple]: + """Feed the raw slices into an in-memory DuckDB and run the denormalization. + Synchronous/CPU-bound — callers offload via to_thread. NEVER attach Postgres from + here: integration tests run inside a rolled-back transaction a second connection + cannot see, and it would add a second credential path.""" + con = duckdb.connect() + try: + for table, ddl in _INPUT_DDL.items(): + con.execute(f"CREATE TABLE {table}({ddl})") + rows = inputs[table] + if rows: + placeholders = ", ".join("?" * len(rows[0])) + con.executemany(f"INSERT INTO {table} VALUES ({placeholders})", rows) + return con.execute(_DENORMALIZE_SQL).fetchall() + finally: + con.close() async def build_workspace_view( db: AsyncSession, *, workspace_id: UUID, offset: int, limit: int ) -> WorkspaceProjection: - """Workspace-level denormalized projection (spec §3.2). Batched: a fixed number of - queries regardless of reference/record count. `offset`/`limit` count references.""" + """Workspace-level denormalized projection (spec §3.2). Postgres serves batched raw + slices — a fixed number of queries regardless of reference/record count — and the + DuckDB statement above does all the semantics. `offset`/`limit` count references.""" schemas = ( (await db.execute(select(Schema).where(Schema.workspace_id == workspace_id).order_by(Schema.name))) .scalars() @@ -682,75 +734,102 @@ async def build_workspace_view( record_rows = ( ( await db.execute( - select(V2Record) - .where(V2Record.schema_id.in_(schema_ids), V2Record.reference.in_(references)) - .order_by(V2Record.inserted_at) + select(V2Record).where( + V2Record.schema_id.in_(schema_ids), V2Record.reference.in_(references) + ) ) ) .scalars() .all() ) - # One effective record per (reference, schema): latest inserted_at wins (§3.1). - effective: dict[tuple[str, UUID], V2Record] = {} - for record in record_rows: - effective[(record.reference, record.schema_id)] = record - record_ids = [r.id for r in effective.values()] - - suggestions: dict[tuple[UUID, UUID], V2Suggestion] = {} - responses: dict[UUID, dict] = {} + record_ids = [r.id for r in record_rows] + + sugg_rows: list[V2Suggestion] = [] + resp_rows: list[V2Response] = [] if record_ids: sugg_rows = ( (await db.execute(select(V2Suggestion).where(V2Suggestion.record_id.in_(record_ids)))) .scalars() .all() ) - suggestions = {(s.record_id, s.question_id): s for s in sugg_rows} - # Latest submitted per record across ALL users — endpoint is user-agnostic and - # cacheable (§3.1); ordering by updated_at lets the dict overwrite older entries. + # Submitted only — drafts must never reach the projection (§3.1). The + # latest-per-record pick happens in SQL (latest_responses window). resp_rows = ( ( await db.execute( - select(V2Response) - .where( + select(V2Response).where( V2Response.record_id.in_(record_ids), V2Response.status == ResponseStatus.submitted, ) - .order_by(V2Response.updated_at) ) ) .scalars() .all() ) - for resp in resp_rows: - responses[resp.record_id] = resp.values or {} - rows: list[WorkspaceProjectionRow] = [] - for reference in references: - rows.extend( - _denormalize_reference( - reference, - schemas=schemas, - questions_by_schema=questions_by_schema, - effective=effective, - suggestions=suggestions, - responses=responses, + question_tuples: list[tuple] = [] + column_tuples: list[tuple] = [] + for schema in schemas: + for question in questions_by_schema.get(schema.id, []): + question_tuples.append( + (str(question.id), str(schema.id), schema.name, question.name, question.type.value) + ) + if question.type == QuestionType.table: + column_tuples.extend((str(question.id), sub) for sub in question.columns) + + inputs = { + "questions": question_tuples, + "question_columns": column_tuples, + "records": [(str(r.id), str(r.schema_id), r.reference, r.inserted_at) for r in record_rows], + "suggestions": [ + ( + str(s.record_id), + str(s.question_id), + json.dumps(s.value), + s.agent, + json.dumps(s.score) if s.score is not None else None, ) + for s in sugg_rows + ], + "responses": [(str(r.record_id), json.dumps(r.values or {}), r.updated_at) for r in resp_rows], + } + long_rows = await to_thread.run_sync(_run_denormalization, inputs) + + # Regroup the ordered long format; SQL's ORDER BY reference matches the page order. + rows: list[WorkspaceProjectionRow] = [] + current: WorkspaceProjectionRow | None = None + for reference, row_index, column_name, value_json, source, record_id, agent, score_json in long_rows: + if current is None or current.reference != reference or current.row_index != row_index: + current = WorkspaceProjectionRow(reference=reference, row_index=row_index, cells={}) + rows.append(current) + if column_name is None: + continue # spine-only row: the reference exists but nothing resolved yet + current.cells[column_name] = WorkspaceProjectionCell( + value=json.loads(value_json), + source=source, + record_id=UUID(record_id), + agent=agent, + score=json.loads(score_json) if score_json is not None else None, ) return WorkspaceProjection(columns=columns, rows=rows, total_references=total_references) ``` -Note: `build_reference_view`'s existing imports already cover most of this; merge rather than duplicate import lines. `AsyncSession`/`UUID` are already imported at the top of the module. +Notes for the implementer: +- `build_reference_view`'s existing imports already cover most of this; merge rather than duplicate import lines. `AsyncSession`/`UUID` are already imported at the top of the module. +- `json_each` is DuckDB's SQLite-style table function (DuckDB ≥1.1) used laterally (`FROM t, json_each(t.col)`). If the installed version rejects the lateral column reference, replace those two spots with the zip-unnest pattern in the SELECT list: `unnest(from_json(col, '["json"]'))` paired with `unnest(generate_series(0, len(...) - 1))` — DuckDB zips parallel unnests. +- All JSON values cross the boundary as strings (`json.dumps` in, `CAST(... AS VARCHAR)` + `json.loads` out) — `score` round-trips `float | list[float]` intact this way. +- Timestamps insert as native `datetime`. If the ORM ever hands back tz-aware datetimes and DuckDB complains, switch the two DDL columns to `TIMESTAMPTZ`. -- [ ] **Step 5: Run to verify pass** +- [ ] **Step 6: Run to verify pass** Run: `cd extralit-server && uv run pytest tests/integration/contexts/v2/test_workspace_projection.py tests/integration/contexts/v2/test_projection.py -v` -Expected: ALL PASS (including the ≤7-statement query-count guard). +Expected: ALL PASS — including the ≤7-statement query-count guard, which still holds because it counts `AsyncSession.execute` calls only; the DuckDB work never touches the session. -- [ ] **Step 6: Commit** +- [ ] **Step 7: Commit** ```bash -git add extralit-server/src/extralit_server/api/schemas/v2/projection.py extralit-server/src/extralit_server/contexts/v2/projection.py extralit-server/tests/integration/contexts/v2/test_workspace_projection.py -git commit -m "feat(server): workspace-level denormalized projection with table fan-out" +git add extralit-server/src/extralit_server/api/schemas/v2/projection.py extralit-server/src/extralit_server/contexts/v2/projection.py extralit-server/tests/integration/contexts/v2/test_workspace_projection.py extralit-server/pyproject.toml extralit-server/uv.lock +git commit -m "feat(server): workspace projection denormalized via in-process DuckDB" ``` --- @@ -1440,7 +1519,7 @@ Expected: all green. (No `npm run build` needed yet — no new build-affecting c - [ ] **Step 4: Scope audit** -Confirm Phase 1 is purely additive: no `package.json` dependency changes, no `nuxt.config.ts`/`vitest.config.ts` changes, no page/component/e2e changes, no deletions. Existing v2 e2e specs (`review-loop`, `draft-lifecycle`, `slashed-reference`, `auth-smoke`, `search-roundtrip`) are untouched and still the active gate. +Confirm Phase 1 is purely additive: no frontend `package.json` dependency changes (the server gained only `duckdb` in Task 3), no `nuxt.config.ts`/`vitest.config.ts` changes, no page/component/e2e changes, no deletions. Existing v2 e2e specs (`review-loop`, `draft-lifecycle`, `slashed-reference`, `auth-smoke`, `search-roundtrip`) are untouched and still the active gate. - [ ] **Step 5: Open PR 1** From 046a3069f5b4bce8266273701428fdc0cd8b5357 Mon Sep 17 00:00:00 2001 From: JonnyTran Date: Mon, 20 Jul 2026 23:59:16 -0700 Subject: [PATCH 05/21] =?UTF-8?q?feat(server):=20accept=20list[dict]=20tab?= =?UTF-8?q?le=20values=20additively=20(spec=20=C2=A73.4)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../extralit_server/validators/v2/values.py | 17 +++++++------ .../tests/unit/validators/v2/test_values.py | 24 +++++++++++++++++++ 2 files changed, 34 insertions(+), 7 deletions(-) diff --git a/extralit-server/src/extralit_server/validators/v2/values.py b/extralit-server/src/extralit_server/validators/v2/values.py index 762479802..3bc48cc19 100644 --- a/extralit-server/src/extralit_server/validators/v2/values.py +++ b/extralit-server/src/extralit_server/validators/v2/values.py @@ -47,14 +47,17 @@ def validate(cls, value, *, type: QuestionType, settings: dict, columns: list[st @staticmethod def _validate_table(value, columns: list[str]) -> None: - if not isinstance(value, dict): - raise UnprocessableEntityError(f"table question expects a dict of values, found {type(value)}") + # Additive contract (spec §3.4): a bare dict is the 1-row case; list[dict] is N rows. + rows = value if isinstance(value, list) else [value] bound = set(columns) - extra = sorted(k for k in value if k not in bound) - if extra: - raise UnprocessableEntityError( - f"table value keys {extra!r} are not bound columns; bound: {sorted(bound)!r}" - ) + for row in rows: + if not isinstance(row, dict): + raise UnprocessableEntityError(f"table question expects a dict of values per row, found {type(row)}") + extra = sorted(k for k in row if k not in bound) + if extra: + raise UnprocessableEntityError( + f"table value keys {extra!r} are not bound columns; bound: {sorted(bound)!r}" + ) class V2SuggestionValidator: diff --git a/extralit-server/tests/unit/validators/v2/test_values.py b/extralit-server/tests/unit/validators/v2/test_values.py index d16686279..40b02f035 100644 --- a/extralit-server/tests/unit/validators/v2/test_values.py +++ b/extralit-server/tests/unit/validators/v2/test_values.py @@ -60,3 +60,27 @@ def test_suggestion_score_length_must_match_list_value(): V2SuggestionValidator.validate( ["yes"], [0.9, 0.1], type=QuestionType.multi_label_selection, settings=MULTI_LABEL_SETTINGS, columns=["c"] ) + + +def test_table_value_accepts_list_of_row_dicts(): + V2ResponseValueValidator.validate( + [{"a": 1}, {"a": 2, "b": "x"}], type=QuestionType.table, settings=TABLE_SETTINGS, columns=["a", "b"] + ) + + +def test_table_value_accepts_empty_list(): + V2ResponseValueValidator.validate([], type=QuestionType.table, settings=TABLE_SETTINGS, columns=["a"]) + + +def test_table_value_list_rejects_unbound_keys_in_any_row(): + with pytest.raises(UnprocessableEntityError, match="not bound"): + V2ResponseValueValidator.validate( + [{"a": 1}, {"z": 2}], type=QuestionType.table, settings=TABLE_SETTINGS, columns=["a", "b"] + ) + + +def test_table_value_list_rejects_non_dict_rows(): + with pytest.raises(UnprocessableEntityError, match="dict of values per row"): + V2ResponseValueValidator.validate( + [{"a": 1}, 5], type=QuestionType.table, settings=TABLE_SETTINGS, columns=["a"] + ) From 5191fcc2dda86f13ab4923496ecdcb1cc0982eb3 Mon Sep 17 00:00:00 2001 From: JonnyTran Date: Tue, 21 Jul 2026 00:06:04 -0700 Subject: [PATCH 06/21] feat(server): enrich ProjectionCell with record_id/agent/score provenance --- .../api/schemas/v2/projection.py | 4 ++++ .../extralit_server/contexts/v2/projection.py | 19 +++++++++++++++---- .../integration/api/v2/test_projection.py | 5 ++++- .../contexts/v2/test_projection.py | 7 ++++++- 4 files changed, 29 insertions(+), 6 deletions(-) diff --git a/extralit-server/src/extralit_server/api/schemas/v2/projection.py b/extralit-server/src/extralit_server/api/schemas/v2/projection.py index bfc901f97..caeb91eb5 100644 --- a/extralit-server/src/extralit_server/api/schemas/v2/projection.py +++ b/extralit-server/src/extralit_server/api/schemas/v2/projection.py @@ -8,6 +8,10 @@ class ProjectionCell(BaseModel): question_name: str value: Any | None = None source: Literal["response", "suggestion"] | None = None # None => neither exists yet + # Enriched provenance (spec §3.2): consumers link and attribute with zero extra calls. + record_id: UUID | None = None + agent: str | None = None + score: float | list[float] | None = None class ProjectionRecord(BaseModel): diff --git a/extralit-server/src/extralit_server/contexts/v2/projection.py b/extralit-server/src/extralit_server/contexts/v2/projection.py index 0b37de044..af1ba7e5b 100644 --- a/extralit-server/src/extralit_server/contexts/v2/projection.py +++ b/extralit-server/src/extralit_server/contexts/v2/projection.py @@ -26,9 +26,9 @@ async def build_reference_view(db: AsyncSession, *, workspace_id: UUID, referenc for q in q_rows: questions_by_schema.setdefault(q.schema_id, []).append(q) - # (record_id, question_id) -> suggestion value + # (record_id, question_id) -> suggestion row sugg_rows = (await db.execute(select(V2Suggestion).where(V2Suggestion.record_id.in_(record_ids)))).scalars().all() - suggestions = {(s.record_id, s.question_id): s.value for s in sugg_rows} + suggestions = {(s.record_id, s.question_id): s for s in sugg_rows} # requesting user's submitted responses only: record_id -> {question_name: value} resp_rows = ( @@ -52,13 +52,24 @@ async def build_reference_view(db: AsyncSession, *, workspace_id: UUID, referenc for question in questions_by_schema.get(record.schema_id, []): wrapped = responses.get(record.id, {}).get(question.name) if wrapped is not None: - cells.append(ProjectionCell(question_name=question.name, value=wrapped.get("value"), source="response")) + cells.append( + ProjectionCell( + question_name=question.name, + value=wrapped.get("value"), + source="response", + record_id=record.id, + ) + ) elif (record.id, question.id) in suggestions: + suggestion = suggestions[(record.id, question.id)] cells.append( ProjectionCell( question_name=question.name, - value=suggestions[(record.id, question.id)], + value=suggestion.value, source="suggestion", + record_id=record.id, + agent=suggestion.agent, + score=suggestion.score, ) ) else: diff --git a/extralit-server/tests/integration/api/v2/test_projection.py b/extralit-server/tests/integration/api/v2/test_projection.py index 9d18b9f36..e1e439cf8 100644 --- a/extralit-server/tests/integration/api/v2/test_projection.py +++ b/extralit-server/tests/integration/api/v2/test_projection.py @@ -28,7 +28,7 @@ async def test_projection_view_resolves_suggestion_cell(async_client, owner_auth workspace = await WorkspaceFactory.create() schema, version, q = await _schema_with_question(workspace) record = await V2RecordFactory.create(version=version, reference="doc-1") - await V2SuggestionFactory.create(record=record, question=q, value="flu") + await V2SuggestionFactory.create(record=record, question=q, value="flu", agent="e2e-agent", score=0.5) resp = await async_client.get( f"/api/v2/projection/references/doc-1?workspace_id={workspace.id}", headers=owner_auth_header @@ -44,6 +44,9 @@ async def test_projection_view_resolves_suggestion_cell(async_client, owner_auth assert cell["question_name"] == "dx" assert cell["value"] == "flu" assert cell["source"] == "suggestion" + assert cell["record_id"] == str(record.id) + assert cell["agent"] == "e2e-agent" + assert cell["score"] == 0.5 async def test_projection_view_unknown_reference_returns_empty(async_client, owner_auth_header): diff --git a/extralit-server/tests/integration/contexts/v2/test_projection.py b/extralit-server/tests/integration/contexts/v2/test_projection.py index b833172b5..5687b2dad 100644 --- a/extralit-server/tests/integration/contexts/v2/test_projection.py +++ b/extralit-server/tests/integration/contexts/v2/test_projection.py @@ -31,12 +31,15 @@ async def _schema_with_question(db): async def test_cell_resolves_to_suggestion_when_no_response(db): schema, version, q = await _schema_with_question(db) record = await V2RecordFactory.create(version=version, reference="doc-1") - await V2SuggestionFactory.create(record=record, question=q, value="flu") + await V2SuggestionFactory.create(record=record, question=q, value="flu", agent="gpt-x", score=0.92) user = await UserFactory.create() view = await projection_ctx.build_reference_view(db, workspace_id=schema.workspace_id, reference="doc-1", user=user) cell = view.records[0].cells[0] assert cell.value == "flu" and cell.source == "suggestion" + assert cell.record_id == record.id + assert cell.agent == "gpt-x" + assert cell.score == 0.92 async def test_cell_resolves_to_response_over_suggestion(db): @@ -51,6 +54,8 @@ async def test_cell_resolves_to_response_over_suggestion(db): view = await projection_ctx.build_reference_view(db, workspace_id=schema.workspace_id, reference="doc-2", user=user) cell = view.records[0].cells[0] assert cell.value == "covid" and cell.source == "response" + assert cell.record_id == record.id + assert cell.agent is None and cell.score is None async def test_cell_is_none_when_no_response_or_suggestion(db): From 15c4e4b3c63fec60992dcc1b658f1c614b853289 Mon Sep 17 00:00:00 2001 From: JonnyTran Date: Tue, 21 Jul 2026 00:19:20 -0700 Subject: [PATCH 07/21] feat(server): workspace projection denormalized via in-process DuckDB --- extralit-server/pyproject.toml | 1 + .../api/schemas/v2/projection.py | 29 ++ .../extralit_server/contexts/v2/projection.py | 325 +++++++++++++++++- .../contexts/v2/test_workspace_projection.py | 204 +++++++++++ extralit-server/uv.lock | 44 +++ 5 files changed, 596 insertions(+), 7 deletions(-) create mode 100644 extralit-server/tests/integration/contexts/v2/test_workspace_projection.py diff --git a/extralit-server/pyproject.toml b/extralit-server/pyproject.toml index 349b21f90..e7018c311 100644 --- a/extralit-server/pyproject.toml +++ b/extralit-server/pyproject.toml @@ -78,6 +78,7 @@ dependencies = [ "opencv-python-headless>=4.11.0.86", "pandera[io]>=0.20", "lancedb>=0.34.0", + "duckdb>=1.5.4", ] [project.optional-dependencies] diff --git a/extralit-server/src/extralit_server/api/schemas/v2/projection.py b/extralit-server/src/extralit_server/api/schemas/v2/projection.py index caeb91eb5..f851ffc62 100644 --- a/extralit-server/src/extralit_server/api/schemas/v2/projection.py +++ b/extralit-server/src/extralit_server/api/schemas/v2/projection.py @@ -25,3 +25,32 @@ class ProjectionView(BaseModel): reference: str records: list[ProjectionRecord] total_records: int + + +class WorkspaceProjectionColumn(BaseModel): + name: str # flat "Schema.question" / "Schema.question.subcol" (spec §3.1) + schema_id: UUID + schema_name: str + question_name: str + sub_column: str | None = None + dtype: str # the question type value; the grid treats it as informational + + +class WorkspaceProjectionCell(BaseModel): + value: Any | None = None + source: Literal["response", "suggestion"] + record_id: UUID + agent: str | None = None + score: float | list[float] | None = None + + +class WorkspaceProjectionRow(BaseModel): + reference: str + row_index: int + cells: dict[str, WorkspaceProjectionCell] # keyed by column name; absent cells omitted + + +class WorkspaceProjection(BaseModel): + columns: list[WorkspaceProjectionColumn] + rows: list[WorkspaceProjectionRow] + total_references: int diff --git a/extralit-server/src/extralit_server/contexts/v2/projection.py b/extralit-server/src/extralit_server/contexts/v2/projection.py index af1ba7e5b..9c5e88d3f 100644 --- a/extralit-server/src/extralit_server/contexts/v2/projection.py +++ b/extralit-server/src/extralit_server/contexts/v2/projection.py @@ -1,16 +1,30 @@ -"""Projection view (spec §17.4): resolve each reviewable cell as -submitted-response(requesting user) -> suggestion, grouped by reference. Query-time, -Postgres-only. A future OLAP materialization can replace this without changing the API.""" +"""Projection views (spec §17.4): resolve each reviewable cell as +submitted-response -> suggestion. `build_reference_view` is the per-reference review form +(Postgres-only, requesting user's responses). `build_workspace_view` is the workspace-wide +denormalized grid: Postgres serves batched raw slices, an in-memory DuckDB does the +denormalization. Both are query-time; a future OLAP materialization can replace them +without changing the API.""" +import json from uuid import UUID -from sqlalchemy import select +import duckdb +from anyio import to_thread +from sqlalchemy import distinct, func, select from sqlalchemy.ext.asyncio import AsyncSession -from extralit_server.api.schemas.v2.projection import ProjectionCell, ProjectionRecord, ProjectionView +from extralit_server.api.schemas.v2.projection import ( + ProjectionCell, + ProjectionRecord, + ProjectionView, + WorkspaceProjection, + WorkspaceProjectionCell, + WorkspaceProjectionColumn, + WorkspaceProjectionRow, +) from extralit_server.contexts.v2 import records as records_ctx -from extralit_server.enums import ResponseStatus -from extralit_server.models.v2 import V2Question, V2Response, V2Suggestion +from extralit_server.enums import QuestionType, ResponseStatus +from extralit_server.models.v2 import Schema, V2Question, V2Record, V2Response, V2Suggestion async def build_reference_view(db: AsyncSession, *, workspace_id: UUID, reference: str, user) -> ProjectionView: @@ -79,3 +93,300 @@ async def build_reference_view(db: AsyncSession, *, workspace_id: UUID, referenc ) return ProjectionView(reference=reference, records=projection_records, total_records=len(records)) + + +def _build_columns( + schemas: list[Schema], + questions_by_schema: dict[UUID, list[V2Question]], +) -> list[WorkspaceProjectionColumn]: + """Flat grid column manifest (spec §3.1): one column per scalar question, one per + table-question sub-column binding, in schema-name then question-definition order.""" + columns: list[WorkspaceProjectionColumn] = [] + for schema in schemas: + for question in questions_by_schema.get(schema.id, []): + if question.type == QuestionType.table: + # sub-columns are the question's `columns` binding (spec §3.4) + for sub in question.columns or []: + columns.append( + WorkspaceProjectionColumn( + name=f"{schema.name}.{question.name}.{sub}", + schema_id=schema.id, + schema_name=schema.name, + question_name=question.name, + sub_column=sub, + dtype=question.type.value, + ) + ) + else: + columns.append( + WorkspaceProjectionColumn( + name=f"{schema.name}.{question.name}", + schema_id=schema.id, + schema_name=schema.name, + question_name=question.name, + sub_column=None, + dtype=question.type.value, + ) + ) + return columns + + +_INPUT_TABLES_DDL = """ +CREATE TABLE questions ( + question_id VARCHAR, schema_id VARCHAR, schema_name VARCHAR, question_name VARCHAR, qtype VARCHAR +); +CREATE TABLE question_columns (question_id VARCHAR, sub_column VARCHAR); +CREATE TABLE records (record_id VARCHAR, schema_id VARCHAR, reference VARCHAR, inserted_at TIMESTAMP); +CREATE TABLE suggestions (record_id VARCHAR, question_id VARCHAR, value_json JSON, agent VARCHAR, score_json JSON); +CREATE TABLE responses (record_id VARCHAR, values_json JSON, updated_at TIMESTAMP); +""" + +_INSERTS = { + "questions": "INSERT INTO questions VALUES (?, ?, ?, ?, ?)", + "question_columns": "INSERT INTO question_columns VALUES (?, ?)", + "records": "INSERT INTO records VALUES (?, ?, ?, ?)", + "suggestions": "INSERT INTO suggestions VALUES (?, ?, ?, ?, ?)", + "responses": "INSERT INTO responses VALUES (?, ?, ?)", +} + +# One statement, one CTE per concern. Emits long-format +# (reference, row_idx, column_name, value_json, source, record_id, agent, score_json) +# ordered so a single linear pass in Python regroups it into rows. +_DENORMALIZE_SQL = """ +WITH effective_records AS ( + -- one effective record per (reference, schema): the latest inserted one + SELECT record_id, schema_id, reference + FROM records + QUALIFY row_number() OVER (PARTITION BY reference, schema_id ORDER BY inserted_at DESC, record_id DESC) = 1 +), +latest_responses AS ( + -- latest submitted response per record, by ANY user (submitted-only filtering happens in Postgres) + SELECT record_id, values_json + FROM responses + WHERE values_json IS NOT NULL + QUALIFY row_number() OVER (PARTITION BY record_id ORDER BY updated_at DESC) = 1 +), +response_keys AS ( + SELECT record_id, values_json, unnest(json_keys(values_json)) AS question_name + FROM latest_responses +), +response_cells AS ( + -- unwrap the {question_name: {"value": ...}} envelope + SELECT record_id, question_name, + json_extract(values_json, '$."' || question_name || '".value') AS value + FROM response_keys +), +resolved AS ( + -- coalesce = response ?? suggestion; (record, question) pairs with neither drop out + SELECT er.reference, + er.record_id, + q.question_id, + q.qtype, + q.schema_name, + q.question_name, + COALESCE(rc.value, s.value_json) AS value, + CASE WHEN rc.value IS NOT NULL THEN 'response' ELSE 'suggestion' END AS source, + CASE WHEN rc.value IS NOT NULL THEN NULL ELSE s.agent END AS agent, + CASE WHEN rc.value IS NOT NULL THEN NULL ELSE s.score_json END AS score + FROM effective_records er + JOIN questions q ON q.schema_id = er.schema_id + LEFT JOIN response_cells rc ON rc.record_id = er.record_id AND rc.question_name = q.question_name + LEFT JOIN suggestions s ON s.record_id = er.record_id AND s.question_id = q.question_id + WHERE COALESCE(rc.value, s.value_json) IS NOT NULL +), +scalar_cells AS ( + SELECT reference, record_id, + schema_name || '.' || question_name AS column_name, + value, source, agent, score + FROM resolved + WHERE qtype <> 'table' AND json_type(value) <> 'NULL' +), +table_arrays AS ( + -- §3.4 normalization: a bare dict is a one-row table + SELECT reference, record_id, question_id, schema_name, question_name, source, agent, score, + CASE WHEN json_type(value) = 'ARRAY' + THEN value + ELSE CAST('[' || CAST(value AS VARCHAR) || ']' AS JSON) + END AS arr + FROM resolved + WHERE qtype = 'table' +), +table_rows AS ( + -- zip-unnest: the index list and the element list are unnested in lockstep + SELECT reference, record_id, question_id, schema_name, question_name, source, agent, score, + unnest(range(CAST(json_array_length(arr) AS BIGINT))) AS row_idx, + unnest(json_extract(arr, '$[*]')) AS row_json + FROM table_arrays +), +table_object_rows AS ( + SELECT * FROM table_rows WHERE json_type(row_json) = 'OBJECT' +), +table_cells AS ( + -- quoting the sub-column into the JSON path handles arbitrary key names + SELECT tr.reference, tr.record_id, tr.row_idx, + tr.schema_name || '.' || tr.question_name || '.' || qc.sub_column AS column_name, + json_extract(tr.row_json, '$."' || qc.sub_column || '"') AS value, + tr.source, tr.agent, tr.score + FROM table_object_rows tr + JOIN question_columns qc ON qc.question_id = tr.question_id + WHERE json_type(json_extract(tr.row_json, '$."' || qc.sub_column || '"')) <> 'NULL' +), +fanout AS ( + SELECT reference, max(row_idx) AS max_idx FROM table_object_rows GROUP BY reference +), +spine AS ( + -- independent stacking: row count = max fan-out across every table on the reference, min 1 + SELECT r.reference, unnest(range(CAST(COALESCE(f.max_idx, 0) + 1 AS BIGINT))) AS row_idx + FROM (SELECT DISTINCT reference FROM records) r + LEFT JOIN fanout f ON f.reference = r.reference +), +all_cells AS ( + -- NULL row_idx = "repeat me onto every spine row" + SELECT reference, CAST(NULL AS BIGINT) AS row_idx, column_name, value, source, record_id, agent, score + FROM scalar_cells + UNION ALL + SELECT reference, row_idx, column_name, value, source, record_id, agent, score + FROM table_cells +) +SELECT s.reference, + s.row_idx, + c.column_name, + CAST(c.value AS VARCHAR) AS value_json, + c.source, + c.record_id, + c.agent, + CAST(c.score AS VARCHAR) AS score_json +FROM spine s +LEFT JOIN all_cells c + ON c.reference = s.reference AND (c.row_idx IS NULL OR c.row_idx = s.row_idx) +ORDER BY s.reference, s.row_idx, c.column_name NULLS LAST +""" + + +def _run_denormalization(inputs: dict[str, list[tuple]]) -> list[tuple]: + """Load the raw Postgres slices into an in-memory DuckDB and run the denormalization. + + Sync and CPU-bound on purpose: callers offload it with `anyio.to_thread.run_sync`. + """ + con = duckdb.connect() + try: + con.execute(_INPUT_TABLES_DDL) + for table, statement in _INSERTS.items(): + rows = inputs.get(table) or [] + if rows: # DuckDB's executemany rejects an empty parameter list + con.executemany(statement, rows) + return con.execute(_DENORMALIZE_SQL).fetchall() + finally: + con.close() + + +async def build_workspace_view(db: AsyncSession, *, workspace_id: UUID, offset: int, limit: int) -> WorkspaceProjection: + """Denormalize a whole workspace into flat grid rows (spec §3). + + Postgres serves batched raw slices only (<=7 statements, independent of the page size); + the in-memory DuckDB statement implements every semantic: effective-record dedup, + response-over-suggestion coalescing, table fan-out with independent stacking and scalar + repetition. `offset`/`limit` count references, not fan-out rows. + """ + schemas = ( + (await db.execute(select(Schema).where(Schema.workspace_id == workspace_id).order_by(Schema.name))) + .scalars() + .all() + ) + if not schemas: + return WorkspaceProjection(columns=[], rows=[], total_references=0) + + schema_ids = [s.id for s in schemas] + schema_names = {s.id: s.name for s in schemas} + questions = ( + ( + await db.execute( + select(V2Question) + .where(V2Question.schema_id.in_(schema_ids)) + .order_by(V2Question.inserted_at, V2Question.name) + ) + ) + .scalars() + .all() + ) + questions_by_schema: dict[UUID, list[V2Question]] = {} + for question in questions: + questions_by_schema.setdefault(question.schema_id, []).append(question) + columns = _build_columns(list(schemas), questions_by_schema) + + total_references = ( + await db.execute(select(func.count(distinct(V2Record.reference))).where(V2Record.schema_id.in_(schema_ids))) + ).scalar_one() + references = ( + ( + await db.execute( + select(V2Record.reference) + .where(V2Record.schema_id.in_(schema_ids)) + .group_by(V2Record.reference) + .order_by(V2Record.reference) + .offset(offset) + .limit(limit) + ) + ) + .scalars() + .all() + ) + if not references: + return WorkspaceProjection(columns=columns, rows=[], total_references=total_references) + + records = ( + ( + await db.execute( + select(V2Record).where(V2Record.schema_id.in_(schema_ids), V2Record.reference.in_(list(references))) + ) + ) + .scalars() + .all() + ) + record_ids = [r.id for r in records] + suggestions = (await db.execute(select(V2Suggestion).where(V2Suggestion.record_id.in_(record_ids)))).scalars().all() + responses = ( + ( + await db.execute( + select(V2Response).where( + V2Response.record_id.in_(record_ids), V2Response.status == ResponseStatus.submitted + ) + ) + ) + .scalars() + .all() + ) + + inputs: dict[str, list[tuple]] = { + "questions": [ + (str(q.id), str(q.schema_id), schema_names[q.schema_id], q.name, q.type.value) for q in questions + ], + "question_columns": [ + (str(q.id), sub) for q in questions if q.type == QuestionType.table for sub in (q.columns or []) + ], + "records": [(str(r.id), str(r.schema_id), r.reference, r.inserted_at) for r in records], + "suggestions": [ + (str(s.record_id), str(s.question_id), json.dumps(s.value), s.agent, json.dumps(s.score)) + for s in suggestions + ], + "responses": [(str(r.record_id), json.dumps(r.values or {}), r.updated_at) for r in responses], + } + output = await to_thread.run_sync(_run_denormalization, inputs) + + rows: list[WorkspaceProjectionRow] = [] + current: WorkspaceProjectionRow | None = None + for reference, row_idx, column_name, value_json, source, record_id, agent, score_json in output: + if current is None or current.reference != reference or current.row_index != row_idx: + current = WorkspaceProjectionRow(reference=reference, row_index=row_idx, cells={}) + rows.append(current) + if column_name is None: # spine-only row: the reference has records but no resolvable cells + continue + current.cells[column_name] = WorkspaceProjectionCell( + value=json.loads(value_json), + source=source, + record_id=UUID(record_id), + agent=agent, + score=json.loads(score_json) if score_json is not None else None, + ) + + return WorkspaceProjection(columns=columns, rows=rows, total_references=total_references) diff --git a/extralit-server/tests/integration/contexts/v2/test_workspace_projection.py b/extralit-server/tests/integration/contexts/v2/test_workspace_projection.py new file mode 100644 index 000000000..981886ddc --- /dev/null +++ b/extralit-server/tests/integration/contexts/v2/test_workspace_projection.py @@ -0,0 +1,204 @@ +import pytest + +from extralit_server.contexts.v2 import projection as projection_ctx +from extralit_server.enums import QuestionType, ResponseStatus +from tests.factories import ( + SchemaFactory, + SchemaVersionFactory, + UserFactory, + V2QuestionFactory, + V2RecordFactory, + V2ResponseFactory, + V2SuggestionFactory, + WorkspaceFactory, +) + +pytestmark = pytest.mark.asyncio + + +async def _make_schema(workspace, name: str): + schema = await SchemaFactory.create(workspace=workspace, name=name) + version = await SchemaVersionFactory.create(schema=schema) + return schema, version + + +async def _add_question(schema, name: str, *, qtype=QuestionType.text, columns=None): + return await V2QuestionFactory.create(schema=schema, name=name, type=qtype, columns=columns or [name]) + + +async def test_columns_manifest_covers_all_schemas_and_fans_out_table_bindings(db): + workspace = await WorkspaceFactory.create() + design, _ = await _make_schema(workspace, "Design") + outcomes, _ = await _make_schema(workspace, "Outcomes") + await _add_question(design, "type") + await _add_question(outcomes, "results", qtype=QuestionType.table, columns=["value", "unit"]) + + view = await projection_ctx.build_workspace_view(db, workspace_id=workspace.id, offset=0, limit=50) + + names = [c.name for c in view.columns] + assert names == ["Design.type", "Outcomes.results.value", "Outcomes.results.unit"] + table_col = view.columns[1] + assert table_col.schema_name == "Outcomes" + assert table_col.question_name == "results" + assert table_col.sub_column == "value" + assert table_col.dtype == "table" + + +async def test_row_universe_is_union_of_references_with_coverage_gaps(db): + workspace = await WorkspaceFactory.create() + design, design_v = await _make_schema(workspace, "Design") + outcomes, outcomes_v = await _make_schema(workspace, "Outcomes") + dq = await _add_question(design, "type") + await _add_question(outcomes, "summary") + rec = await V2RecordFactory.create(version=design_v, reference="10.1/a") + await V2SuggestionFactory.create(record=rec, question=dq, value="RCT") + await V2RecordFactory.create(version=outcomes_v, reference="10.1/b") + + view = await projection_ctx.build_workspace_view(db, workspace_id=workspace.id, offset=0, limit=50) + + assert view.total_references == 2 + assert [(r.reference, r.row_index) for r in view.rows] == [("10.1/a", 0), ("10.1/b", 0)] + row_a, row_b = view.rows + assert row_a.cells["Design.type"].value == "RCT" + assert "Outcomes.summary" not in row_a.cells # no Outcomes record: coverage gap, cell omitted + assert row_b.cells == {} # record exists but neither response nor suggestion + + +async def test_latest_submitted_response_any_user_beats_suggestion(db): + workspace = await WorkspaceFactory.create() + schema, version = await _make_schema(workspace, "Design") + q = await _add_question(schema, "type") + rec = await V2RecordFactory.create(version=version, reference="10.1/a") + await V2SuggestionFactory.create(record=rec, question=q, value="cohort", agent="gpt-x", score=0.9) + user1 = await UserFactory.create() + user2 = await UserFactory.create() + await V2ResponseFactory.create( + record=rec, user=user1, values={"type": {"value": "RCT-old"}}, status=ResponseStatus.submitted + ) + await V2ResponseFactory.create( + record=rec, user=user2, values={"type": {"value": "RCT"}}, status=ResponseStatus.submitted + ) + + view = await projection_ctx.build_workspace_view(db, workspace_id=workspace.id, offset=0, limit=50) + + cell = view.rows[0].cells["Design.type"] + assert cell.value == "RCT" # later updated_at wins across users + assert cell.source == "response" + assert cell.record_id == rec.id + assert cell.agent is None and cell.score is None + + +async def test_draft_responses_never_appear(db): + workspace = await WorkspaceFactory.create() + schema, version = await _make_schema(workspace, "Design") + q = await _add_question(schema, "type") + rec = await V2RecordFactory.create(version=version, reference="10.1/a") + await V2SuggestionFactory.create(record=rec, question=q, value="cohort", agent="gpt-x", score=0.9) + user = await UserFactory.create() + await V2ResponseFactory.create( + record=rec, user=user, values={"type": {"value": "draft-val"}}, status=ResponseStatus.draft + ) + + view = await projection_ctx.build_workspace_view(db, workspace_id=workspace.id, offset=0, limit=50) + + cell = view.rows[0].cells["Design.type"] + assert cell.value == "cohort" + assert cell.source == "suggestion" + assert cell.agent == "gpt-x" + assert cell.score == 0.9 + + +async def test_table_fanout_independent_stacking_and_scalar_repetition(db): + workspace = await WorkspaceFactory.create() + schema, version = await _make_schema(workspace, "Outcomes") + scalar_q = await _add_question(schema, "design") + t1 = await _add_question(schema, "results", qtype=QuestionType.table, columns=["value", "unit"]) + t2 = await _add_question(schema, "arms", qtype=QuestionType.table, columns=["arm"]) + rec = await V2RecordFactory.create(version=version, reference="10.1/a") + await V2SuggestionFactory.create(record=rec, question=scalar_q, value="RCT") + await V2SuggestionFactory.create( + record=rec, + question=t1, + value=[{"value": "12%", "unit": "pct"}, {"value": "8%", "unit": "pct"}, {"value": "3%"}], + ) + await V2SuggestionFactory.create(record=rec, question=t2, value=[{"arm": "control"}, {"arm": "treated"}]) + + view = await projection_ctx.build_workspace_view(db, workspace_id=workspace.id, offset=0, limit=50) + + assert view.total_references == 1 + assert len(view.rows) == 3 # max(3, 2), NOT 3*2 (no cartesian product) + assert [r.row_index for r in view.rows] == [0, 1, 2] + # scalars repeat on every fan-out row (true denormalized rows) + assert all(r.cells["Outcomes.design"].value == "RCT" for r in view.rows) + assert [r.cells["Outcomes.results.value"].value for r in view.rows] == ["12%", "8%", "3%"] + # shorter table just ends (independent stacking): row 2 has no arms cell + assert [r.cells.get("Outcomes.arms.arm") and r.cells["Outcomes.arms.arm"].value for r in view.rows] == [ + "control", + "treated", + None, + ] + # missing sub-key on a row dict is omitted + assert "Outcomes.results.unit" not in view.rows[2].cells + + +async def test_single_dict_table_value_is_one_row(db): + workspace = await WorkspaceFactory.create() + schema, version = await _make_schema(workspace, "Outcomes") + t = await _add_question(schema, "results", qtype=QuestionType.table, columns=["value"]) + rec = await V2RecordFactory.create(version=version, reference="10.1/a") + await V2SuggestionFactory.create(record=rec, question=t, value={"value": "12%"}) + + view = await projection_ctx.build_workspace_view(db, workspace_id=workspace.id, offset=0, limit=50) + + assert len(view.rows) == 1 + assert view.rows[0].cells["Outcomes.results.value"].value == "12%" + + +async def test_effective_record_is_latest_inserted_per_reference_schema(db): + workspace = await WorkspaceFactory.create() + schema, version = await _make_schema(workspace, "Design") + q = await _add_question(schema, "type") + old = await V2RecordFactory.create(version=version, reference="10.1/a") + new = await V2RecordFactory.create(version=version, reference="10.1/a") + await V2SuggestionFactory.create(record=old, question=q, value="old") + await V2SuggestionFactory.create(record=new, question=q, value="new") + + view = await projection_ctx.build_workspace_view(db, workspace_id=workspace.id, offset=0, limit=50) + + assert len(view.rows) == 1 + assert view.rows[0].cells["Design.type"].value == "new" + assert view.rows[0].cells["Design.type"].record_id == new.id + + +async def test_pagination_counts_references_not_rows(db): + workspace = await WorkspaceFactory.create() + schema, version = await _make_schema(workspace, "Design") + await _add_question(schema, "type") + for i in range(5): + await V2RecordFactory.create(version=version, reference=f"10.1/{i}") + + page = await projection_ctx.build_workspace_view(db, workspace_id=workspace.id, offset=2, limit=2) + + assert page.total_references == 5 + assert [r.reference for r in page.rows] == ["10.1/2", "10.1/3"] # ordered by reference + + +async def test_query_count_is_constant_regardless_of_reference_count(db, monkeypatch): + workspace = await WorkspaceFactory.create() + schema, version = await _make_schema(workspace, "Design") + q = await _add_question(schema, "type") + for i in range(6): + rec = await V2RecordFactory.create(version=version, reference=f"10.1/{i}") + await V2SuggestionFactory.create(record=rec, question=q, value=f"v{i}") + + executed: list[object] = [] + original_execute = db.execute + + async def counting_execute(*args, **kwargs): + executed.append(args[0]) + return await original_execute(*args, **kwargs) + + monkeypatch.setattr(db, "execute", counting_execute) + await projection_ctx.build_workspace_view(db, workspace_id=workspace.id, offset=0, limit=50) + # schemas, questions, ref-count, ref-page, records, suggestions, responses => 7 max + assert len(executed) <= 7, f"N+1 regression: {len(executed)} statements" diff --git a/extralit-server/uv.lock b/extralit-server/uv.lock index 3fac23957..9db523964 100644 --- a/extralit-server/uv.lock +++ b/extralit-server/uv.lock @@ -1088,6 +1088,48 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/12/b3/231ffd4ab1fc9d679809f356cebee130ac7daa00d6d6f3206dd4fd137e9e/distro-1.9.0-py3-none-any.whl", hash = "sha256:7bffd925d65168f85027d8da9af6bddab658135b840670a223589bc0c8ef02b2", size = 20277, upload-time = "2023-12-24T09:54:30.421Z" }, ] +[[package]] +name = "duckdb" +version = "1.5.4" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/31/29/9bad86ed7aa812d8c822a27c15c355b6d5423b991feeec86ed18027b6daa/duckdb-1.5.4.tar.gz", hash = "sha256:f9e32f1cdd106793d79d190186bed9e75289d51e68bd9174e47c04bffedeab6f", size = 18046634, upload-time = "2026-06-17T10:48:52.499Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/3c/ee/69340af74a3aa21838f14c16a0dd3e58461896ccba41f6bc7f0a01536e23/duckdb-1.5.4-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:3ddd9533ce80f9b851bdd6276960a9286166514a9ceca43d5bc2f0d5842c490d", size = 32656177, upload-time = "2026-06-17T10:47:31.044Z" }, + { url = "https://files.pythonhosted.org/packages/73/18/9da267ade389d4e7e533ac0c77b3a7041513a66efab93beb84f27627b0b8/duckdb-1.5.4-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:360f2542d09759c3739400f8b787e29b43ba0da665c21756216291458bf6fc59", size = 17318966, upload-time = "2026-06-17T10:47:33.688Z" }, + { url = "https://files.pythonhosted.org/packages/76/b4/ad73c1a396288e443b18af50819448060b318c1e933305167c1d7f98a507/duckdb-1.5.4-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:0cf932055061e544d3fa27cc6c147da25f3f681ee5980157fb55e77d6c2d9c63", size = 15467572, upload-time = "2026-06-17T10:47:36.071Z" }, + { url = "https://files.pythonhosted.org/packages/6f/06/2c52ce3b97c3f21111f3c98a2121ed002e33f86488f55098a37825af6d4a/duckdb-1.5.4-cp310-cp310-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c2d58d39f5e65419cdc27e3875cba4a729a3bbf6bf4016aefb4a2a65335a1d42", size = 19344044, upload-time = "2026-06-17T10:47:38.174Z" }, + { url = "https://files.pythonhosted.org/packages/5a/7d/5c0cc66fb90a1b14474eebb5ab535eacc51cb20b0e45358348b51c07abc9/duckdb-1.5.4-cp310-cp310-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a9a10dc40469b9c0e458625d2a8359461a982c6151bb53ff259fea00c4695ad4", size = 21448215, upload-time = "2026-06-17T10:47:40.617Z" }, + { url = "https://files.pythonhosted.org/packages/3c/2c/16c3ea201855cdfb7dde52ea3678d0536861cb485ffad46cf345436d658d/duckdb-1.5.4-cp310-cp310-win_amd64.whl", hash = "sha256:3565550adbf160ef7a2ee3395470570182f11233983ad818bd7d5f9e349f92b2", size = 13132138, upload-time = "2026-06-17T10:47:43.085Z" }, + { url = "https://files.pythonhosted.org/packages/56/bb/7921dabd50daef3969f14cd8a5a14c24eee337db7914a462f2defa8add92/duckdb-1.5.4-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:3fb41d9cfccb7e44511eeeed263ae98143ca63bdb1ef84631ba637c314efa1b5", size = 32663142, upload-time = "2026-06-17T10:47:45.471Z" }, + { url = "https://files.pythonhosted.org/packages/a6/83/2137765eaba6a9aefe3bb9848ddaac7407fe3ba19b292f98b31f3b7ab27f/duckdb-1.5.4-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:8ba7b666bc9c78d6a930ee9f469024149f0c6a23fb7d2c3418aad6774339bec0", size = 17321485, upload-time = "2026-06-17T10:47:47.778Z" }, + { url = "https://files.pythonhosted.org/packages/0e/b2/a02c1ee43fd7e8cf1fc2e3d377f3dcf9d4a3e58a4549557516e1866ff0da/duckdb-1.5.4-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:9d9e6817fcbc09d2605a2c8c041ac7824d738d917c35a4d427e977647e1d7944", size = 15470820, upload-time = "2026-06-17T10:47:49.977Z" }, + { url = "https://files.pythonhosted.org/packages/d8/48/a243d30223b024bc6057abe472b002cff01e97efefb4d2f0b0dcc5aece0b/duckdb-1.5.4-cp311-cp311-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:02dd9f9a6124069213f13e3a474c208028c472fe1acdae12b38761f954fe4fc6", size = 19341849, upload-time = "2026-06-17T10:47:52.205Z" }, + { url = "https://files.pythonhosted.org/packages/08/ff/a5d48de4771e2403a8ef26a20dc7457b1c8f7e398ff0caf9c0cad8805f89/duckdb-1.5.4-cp311-cp311-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ccc7f2694d02b4763fee61021d45e12f7bc5743993686563957df0cef799fbae", size = 21451698, upload-time = "2026-06-17T10:47:54.653Z" }, + { url = "https://files.pythonhosted.org/packages/79/b8/8244d7741b4afae67775cf0cb0d4eb9e923a83110907e4801e17fa078480/duckdb-1.5.4-cp311-cp311-win_amd64.whl", hash = "sha256:4c430e788d99b50854209bf2833ba36a45df75e57f86efb477046cd408bbd077", size = 13132643, upload-time = "2026-06-17T10:47:56.75Z" }, + { url = "https://files.pythonhosted.org/packages/e4/57/8169822a37f6dd7d561c567f9007e3cf04bf97bccb619afe90db849c0962/duckdb-1.5.4-cp311-cp311-win_arm64.whl", hash = "sha256:e2dc8340cfb6006025a798c50f40126d6e945a1d2487be94667bb4166556ce7b", size = 13986386, upload-time = "2026-06-17T10:47:59.345Z" }, + { url = "https://files.pythonhosted.org/packages/c8/f2/e2f4b477ae3a3b40e8b5f429832e48edb62ed9da99807cc4902e157e5646/duckdb-1.5.4-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:291a9e7502551170af989ff63139a7a49e99d68edbc5ef5017ac27541fe54c65", size = 32708876, upload-time = "2026-06-17T10:48:01.527Z" }, + { url = "https://files.pythonhosted.org/packages/2e/2b/b698d82a5e1e30b6a05748d72045f672994c6b22f4f0f8423523608b991f/duckdb-1.5.4-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:83e8c089bbb756ca4471d8b05943b80a106058697cf00615e70423106bb783bc", size = 17346125, upload-time = "2026-06-17T10:48:04.035Z" }, + { url = "https://files.pythonhosted.org/packages/71/75/37e13f39268eaf34864453b3a039c4a1ff0b088d3eae45a4289b41c98c1b/duckdb-1.5.4-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:ff96d2a342b200e1ec6f1f19986c77f4ac16a49b6112f71c5b763989203a9d60", size = 15488133, upload-time = "2026-06-17T10:48:06.312Z" }, + { url = "https://files.pythonhosted.org/packages/cc/59/2d082af578f689231798245b54562c61416e49049b0bda81a06c56a4b53e/duckdb-1.5.4-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8f935ef210ab00bc94bb1e3052697adaa36bb0ce7bdfeda8b0f34e2ff1643870", size = 19367895, upload-time = "2026-06-17T10:48:08.59Z" }, + { url = "https://files.pythonhosted.org/packages/52/2b/55c34d2863a76ca824ef8274691e84240b4ff1acde3d231709e82557c240/duckdb-1.5.4-cp312-cp312-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0cda263d8c20addb8d4f95464787cbe0af1144f7ab7e21db3709fb826ee01725", size = 21486499, upload-time = "2026-06-17T10:48:10.963Z" }, + { url = "https://files.pythonhosted.org/packages/cf/30/ade5952b8182fac86fab43b95ebe3836e66381d0ad64eb1e54bd8207c988/duckdb-1.5.4-cp312-cp312-win_amd64.whl", hash = "sha256:266c7c909558ce7377f57d082cee408aadebdd9111be017558ca54e44a031037", size = 13147934, upload-time = "2026-06-17T10:48:13.061Z" }, + { url = "https://files.pythonhosted.org/packages/f5/00/278f0f70e25b9911afe2fd227b9460f2e6d76177f0dcc03f7f1454afefa5/duckdb-1.5.4-cp312-cp312-win_arm64.whl", hash = "sha256:f14e79a006341f29ee5a2692a24dac5114e77533d579c57ec39124adf0135033", size = 13965235, upload-time = "2026-06-17T10:48:15.782Z" }, + { url = "https://files.pythonhosted.org/packages/da/69/3fcb34e523a9bad1f0557a6c7691a71ba66c43a05e5be9ee96a9a841ed65/duckdb-1.5.4-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:42a612e67d64450b446eb69695290d460713eef46e0f64467ab9dfe96264ee05", size = 32708366, upload-time = "2026-06-17T10:48:18.084Z" }, + { url = "https://files.pythonhosted.org/packages/f5/5f/bff5054c2c1d65decab36aa6296621e51a2a575a9f250db7ab9b83a325d6/duckdb-1.5.4-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:3fb6f07d54ecf4d0d3c5179a2361fdddfafa14de4fc42696de4632479b703421", size = 17345735, upload-time = "2026-06-17T10:48:20.67Z" }, + { url = "https://files.pythonhosted.org/packages/93/12/d1b2b344e9699246aada6f9de5156e708fb476e2780e5bff9b5d95fe11d9/duckdb-1.5.4-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:0f32ad7e0286c1c29ab6c73b29118c86101f8eee46aae54f54d0b50916f542f6", size = 15488568, upload-time = "2026-06-17T10:48:23.038Z" }, + { url = "https://files.pythonhosted.org/packages/c1/d1/ac56c6096e3e95da60b2c5dd5a0f0eb5540a80622e2e4f8faab893ec4e96/duckdb-1.5.4-cp313-cp313-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:698ec90bd5d5538bd5f6d212a4b61af443d240703cf45f134738535026556ea5", size = 19368184, upload-time = "2026-06-17T10:48:25.601Z" }, + { url = "https://files.pythonhosted.org/packages/1e/0b/2ae4c3e157a19d9b4ac1f09a5dea6f93012334cc2db09f1e0c71eb99693d/duckdb-1.5.4-cp313-cp313-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:136cea7f886b78caf4035485b4b1e766e8b309e999f9e83a966f81ebb8122844", size = 21486523, upload-time = "2026-06-17T10:48:27.817Z" }, + { url = "https://files.pythonhosted.org/packages/64/7b/c3d8d21e0d0db8faa81eeeb3a55b9932f5a0a16466cb968dc713a653d701/duckdb-1.5.4-cp313-cp313-win_amd64.whl", hash = "sha256:bd6777e8ddd74fb603a6d09766bfcff28638189f8aaa61fc0dffd9e9a4baa8e5", size = 13147807, upload-time = "2026-06-17T10:48:30.017Z" }, + { url = "https://files.pythonhosted.org/packages/44/48/ddf8d3740e3d28582944f70d84e720b5dc28c10ec22b668a0e0bd965f2f2/duckdb-1.5.4-cp313-cp313-win_arm64.whl", hash = "sha256:73f4878a3012283024a64a1909e440aac12091ef336f671fc142f7e87449ce0c", size = 13965189, upload-time = "2026-06-17T10:48:32.251Z" }, + { url = "https://files.pythonhosted.org/packages/62/01/67ac4cbc8e552a1e14c029b5c443d828e68f94d5d913c574f577e1db277e/duckdb-1.5.4-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:4647968629d0677bbcc2416c7aeda8685eb84e4ca15a6dbd4f82a66cfc91a532", size = 32714364, upload-time = "2026-06-17T10:48:34.724Z" }, + { url = "https://files.pythonhosted.org/packages/4e/0e/eb44d983fa56b175f971eea251bde284a36d26cbb93fcb68035061f54078/duckdb-1.5.4-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:e8fcef301cf68d3951ea1eb8ac4d76cea0a6f6a08f4c78fe4026fc96d217bebc", size = 17349820, upload-time = "2026-06-17T10:48:37.126Z" }, + { url = "https://files.pythonhosted.org/packages/10/b2/b9dc7624b105d414585b8530451c1162c0b4750c0be9be2e497bb47a8a9b/duckdb-1.5.4-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:f6f39cd0dc6948dee17fd130aec55114f97a8ef6e1db519b9774087962bc5c8c", size = 15498160, upload-time = "2026-06-17T10:48:40.032Z" }, + { url = "https://files.pythonhosted.org/packages/b7/57/61356444f6a8c62dec3c3d129abfc53f428de1d484093d1bb381db441231/duckdb-1.5.4-cp314-cp314-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:262f068158beb5943f2c618f4e54b46db8306b959f90dce956f90a89f613673d", size = 19374183, upload-time = "2026-06-17T10:48:42.698Z" }, + { url = "https://files.pythonhosted.org/packages/b0/f4/d5d633dd7c5138d8f7c434e6ac2553c831b7fb658494efa8d0bc73df8623/duckdb-1.5.4-cp314-cp314-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:4d2307a76d199077b0055b354e90e857479461a0d875437535dd4833172c8b6d", size = 21487202, upload-time = "2026-06-17T10:48:45.407Z" }, + { url = "https://files.pythonhosted.org/packages/c0/26/5be13bbd5c3421dccfc1ad4ca9da4b97c5a3ddd73f66542092f3167ec52c/duckdb-1.5.4-cp314-cp314-win_amd64.whl", hash = "sha256:6dcbb81a1276bc48deb4d562bce4f8895e4fc6348750a096e30052345c6d6552", size = 13666989, upload-time = "2026-06-17T10:48:47.764Z" }, + { url = "https://files.pythonhosted.org/packages/dc/82/4d52f3f9f9703a226b26b80bdae3f6905aeefe5221bf1815fc93ff02ca25/duckdb-1.5.4-cp314-cp314-win_arm64.whl", hash = "sha256:0f8722346024e5d9f02b58bf7b0491a629f97fdc8a04a10e432940f471ee387a", size = 14449863, upload-time = "2026-06-17T10:48:50.18Z" }, +] + [[package]] name = "ecdsa" version = "0.19.1" @@ -1164,6 +1206,7 @@ dependencies = [ { name = "brotli-asgi" }, { name = "click" }, { name = "datasets" }, + { name = "duckdb" }, { name = "elasticsearch8", extra = ["async"] }, { name = "fastapi" }, { name = "filelock" }, @@ -1240,6 +1283,7 @@ requires-dist = [ { name = "brotli-asgi", specifier = "~=1.4.0" }, { name = "click", specifier = ">=8.2.0" }, { name = "datasets", specifier = ">=3.6.0" }, + { name = "duckdb", specifier = ">=1.5.4" }, { name = "elasticsearch8", extras = ["async"], specifier = "~=8.7.0" }, { name = "fastapi", specifier = "~=0.115.0" }, { name = "filelock", specifier = ">=3.25.2" }, From 4225c7ba34ed7392b3b659b4090868e99d1d1bd6 Mon Sep 17 00:00:00 2001 From: JonnyTran Date: Tue, 21 Jul 2026 11:25:14 -0700 Subject: [PATCH 08/21] fix(server): escape JSON paths in workspace projection denormalization --- .../extralit_server/contexts/v2/projection.py | 56 ++++++++++++++----- .../contexts/v2/test_workspace_projection.py | 28 ++++++++++ 2 files changed, 71 insertions(+), 13 deletions(-) diff --git a/extralit-server/src/extralit_server/contexts/v2/projection.py b/extralit-server/src/extralit_server/contexts/v2/projection.py index 9c5e88d3f..ff3b2b64d 100644 --- a/extralit-server/src/extralit_server/contexts/v2/projection.py +++ b/extralit-server/src/extralit_server/contexts/v2/projection.py @@ -131,11 +131,24 @@ def _build_columns( return columns +def _json_path_escape(key: str) -> str: + """Escape a key for interpolation inside a double-quoted DuckDB JSON path segment. + + Question names and table sub-column bindings are unconstrained user input, and DuckDB parses + the JSON path as a whole: an unescaped `"` raises `JSON path error near ...`, which fails the + entire workspace grid rather than just the offending cell. + """ + return key.replace("\\", "\\\\").replace('"', '\\"') + + _INPUT_TABLES_DDL = """ CREATE TABLE questions ( question_id VARCHAR, schema_id VARCHAR, schema_name VARCHAR, question_name VARCHAR, qtype VARCHAR ); -CREATE TABLE question_columns (question_id VARCHAR, sub_column VARCHAR); +-- `sub_path` is `sub_column` pre-escaped for interpolation into a quoted JSON path +-- (see `_json_path_escape`); a raw `"` or `\` in the binding would otherwise be a bind-time +-- JSON-path parse error that fails the whole statement, not just that cell. +CREATE TABLE question_columns (question_id VARCHAR, sub_column VARCHAR, sub_path VARCHAR); CREATE TABLE records (record_id VARCHAR, schema_id VARCHAR, reference VARCHAR, inserted_at TIMESTAMP); CREATE TABLE suggestions (record_id VARCHAR, question_id VARCHAR, value_json JSON, agent VARCHAR, score_json JSON); CREATE TABLE responses (record_id VARCHAR, values_json JSON, updated_at TIMESTAMP); @@ -143,7 +156,7 @@ def _build_columns( _INSERTS = { "questions": "INSERT INTO questions VALUES (?, ?, ?, ?, ?)", - "question_columns": "INSERT INTO question_columns VALUES (?, ?)", + "question_columns": "INSERT INTO question_columns VALUES (?, ?, ?)", "records": "INSERT INTO records VALUES (?, ?, ?, ?)", "suggestions": "INSERT INTO suggestions VALUES (?, ?, ?, ?, ?)", "responses": "INSERT INTO responses VALUES (?, ?, ?)", @@ -160,21 +173,27 @@ def _build_columns( QUALIFY row_number() OVER (PARTITION BY reference, schema_id ORDER BY inserted_at DESC, record_id DESC) = 1 ), latest_responses AS ( - -- latest submitted response per record, by ANY user (submitted-only filtering happens in Postgres) + -- Record-level selection, intentional per spec §3.2: exactly ONE response *envelope* wins + -- per record -- the latest submitted one by ANY user (submitted-only filtering happens in + -- Postgres). A question absent from that envelope therefore falls back to its suggestion + -- even when an earlier submitted response answered it; envelopes are not merged per cell. SELECT record_id, values_json FROM responses WHERE values_json IS NOT NULL QUALIFY row_number() OVER (PARTITION BY record_id ORDER BY updated_at DESC) = 1 ), -response_keys AS ( - SELECT record_id, values_json, unnest(json_keys(values_json)) AS question_name +response_entries AS ( + -- zip-unnest the envelope: `json_keys` and the `'$.*'` wildcard walk the object in the same + -- order, which avoids interpolating a data-derived key into a JSON path (unescapable here). + SELECT record_id, + unnest(json_keys(values_json)) AS question_name, + unnest(json_extract(values_json, '$.*')) AS entry FROM latest_responses ), response_cells AS ( -- unwrap the {question_name: {"value": ...}} envelope - SELECT record_id, question_name, - json_extract(values_json, '$."' || question_name || '".value') AS value - FROM response_keys + SELECT record_id, question_name, json_extract(entry, '$.value') AS value + FROM response_entries ), resolved AS ( -- coalesce = response ?? suggestion; (record, question) pairs with neither drop out @@ -221,15 +240,18 @@ def _build_columns( table_object_rows AS ( SELECT * FROM table_rows WHERE json_type(row_json) = 'OBJECT' ), -table_cells AS ( - -- quoting the sub-column into the JSON path handles arbitrary key names +table_cell_values AS ( + -- the pre-escaped `sub_path` quoted into the JSON path handles arbitrary key names SELECT tr.reference, tr.record_id, tr.row_idx, tr.schema_name || '.' || tr.question_name || '.' || qc.sub_column AS column_name, - json_extract(tr.row_json, '$."' || qc.sub_column || '"') AS value, + json_extract(tr.row_json, '$."' || qc.sub_path || '"') AS value, tr.source, tr.agent, tr.score FROM table_object_rows tr JOIN question_columns qc ON qc.question_id = tr.question_id - WHERE json_type(json_extract(tr.row_json, '$."' || qc.sub_column || '"')) <> 'NULL' +), +table_cells AS ( + -- absent sub-key (SQL NULL) and JSON-null alike are omitted + SELECT * FROM table_cell_values WHERE json_type(value) <> 'NULL' ), fanout AS ( SELECT reference, max(row_idx) AS max_idx FROM table_object_rows GROUP BY reference @@ -287,6 +309,11 @@ async def build_workspace_view(db: AsyncSession, *, workspace_id: UUID, offset: the in-memory DuckDB statement implements every semantic: effective-record dedup, response-over-suggestion coalescing, table fan-out with independent stacking and scalar repetition. `offset`/`limit` count references, not fan-out rows. + + Coalescing is record-level, intentionally (spec §3.2): the latest submitted response + *envelope* per record wins outright, across all users. A question the winning envelope does + not contain falls back to its suggestion even if an earlier submitted response answered it + — envelopes are never merged cell-by-cell. """ schemas = ( (await db.execute(select(Schema).where(Schema.workspace_id == workspace_id).order_by(Schema.name))) @@ -362,7 +389,10 @@ async def build_workspace_view(db: AsyncSession, *, workspace_id: UUID, offset: (str(q.id), str(q.schema_id), schema_names[q.schema_id], q.name, q.type.value) for q in questions ], "question_columns": [ - (str(q.id), sub) for q in questions if q.type == QuestionType.table for sub in (q.columns or []) + (str(q.id), sub, _json_path_escape(sub)) + for q in questions + if q.type == QuestionType.table + for sub in (q.columns or []) ], "records": [(str(r.id), str(r.schema_id), r.reference, r.inserted_at) for r in records], "suggestions": [ diff --git a/extralit-server/tests/integration/contexts/v2/test_workspace_projection.py b/extralit-server/tests/integration/contexts/v2/test_workspace_projection.py index 981886ddc..e2e2a8ee0 100644 --- a/extralit-server/tests/integration/contexts/v2/test_workspace_projection.py +++ b/extralit-server/tests/integration/contexts/v2/test_workspace_projection.py @@ -154,6 +154,34 @@ async def test_single_dict_table_value_is_one_row(db): assert view.rows[0].cells["Outcomes.results.value"].value == "12%" +async def test_quotes_and_backslashes_in_names_do_not_break_json_paths(db): + """Question names and sub-column bindings are unconstrained user input; a raw `"` used to be + a bind-time JSON-path error that failed the whole grid, not just the offending cell.""" + workspace = await WorkspaceFactory.create() + schema, version = await _make_schema(workspace, "Outcomes") + await _add_question(schema, 'we"ird\\name') + t = await _add_question(schema, "results", qtype=QuestionType.table, columns=['sub"col', "back\\slash"]) + rec = await V2RecordFactory.create(version=version, reference="10.1/a") + await V2SuggestionFactory.create(record=rec, question=t, value=[{'sub"col': "A", "back\\slash": "B"}]) + user = await UserFactory.create() + await V2ResponseFactory.create( + record=rec, user=user, values={'we"ird\\name': {"value": "RCT"}}, status=ResponseStatus.submitted + ) + + view = await projection_ctx.build_workspace_view(db, workspace_id=workspace.id, offset=0, limit=50) + + assert [c.name for c in view.columns] == [ + 'Outcomes.we"ird\\name', + 'Outcomes.results.sub"col', + "Outcomes.results.back\\slash", + ] + cells = view.rows[0].cells + assert cells['Outcomes.we"ird\\name'].value == "RCT" + assert cells['Outcomes.we"ird\\name'].source == "response" + assert cells['Outcomes.results.sub"col'].value == "A" + assert cells["Outcomes.results.back\\slash"].value == "B" + + async def test_effective_record_is_latest_inserted_per_reference_schema(db): workspace = await WorkspaceFactory.create() schema, version = await _make_schema(workspace, "Design") From de18a6f3e3c64362568aaee4a65737ec0240f622 Mon Sep 17 00:00:00 2001 From: JonnyTran Date: Tue, 21 Jul 2026 11:32:40 -0700 Subject: [PATCH 09/21] fix(server): join sub-column bindings instead of interpolating JSON paths --- .../extralit_server/contexts/v2/projection.py | 51 ++++++++----------- .../contexts/v2/test_workspace_projection.py | 20 ++++++-- 2 files changed, 37 insertions(+), 34 deletions(-) diff --git a/extralit-server/src/extralit_server/contexts/v2/projection.py b/extralit-server/src/extralit_server/contexts/v2/projection.py index ff3b2b64d..96ec9610a 100644 --- a/extralit-server/src/extralit_server/contexts/v2/projection.py +++ b/extralit-server/src/extralit_server/contexts/v2/projection.py @@ -131,24 +131,15 @@ def _build_columns( return columns -def _json_path_escape(key: str) -> str: - """Escape a key for interpolation inside a double-quoted DuckDB JSON path segment. - - Question names and table sub-column bindings are unconstrained user input, and DuckDB parses - the JSON path as a whole: an unescaped `"` raises `JSON path error near ...`, which fails the - entire workspace grid rather than just the offending cell. - """ - return key.replace("\\", "\\\\").replace('"', '\\"') - - _INPUT_TABLES_DDL = """ CREATE TABLE questions ( question_id VARCHAR, schema_id VARCHAR, schema_name VARCHAR, question_name VARCHAR, qtype VARCHAR ); --- `sub_path` is `sub_column` pre-escaped for interpolation into a quoted JSON path --- (see `_json_path_escape`); a raw `"` or `\` in the binding would otherwise be a bind-time --- JSON-path parse error that fails the whole statement, not just that cell. -CREATE TABLE question_columns (question_id VARCHAR, sub_column VARCHAR, sub_path VARCHAR); +-- Sub-column bindings are unconstrained user input and are never interpolated into a JSON path: +-- the statement joins them against unnested object keys instead. A quote, a backslash or an +-- empty name in a path aborts the whole statement at execution time (not just that cell), and a +-- name of '*' would silently match every key. +CREATE TABLE question_columns (question_id VARCHAR, sub_column VARCHAR); CREATE TABLE records (record_id VARCHAR, schema_id VARCHAR, reference VARCHAR, inserted_at TIMESTAMP); CREATE TABLE suggestions (record_id VARCHAR, question_id VARCHAR, value_json JSON, agent VARCHAR, score_json JSON); CREATE TABLE responses (record_id VARCHAR, values_json JSON, updated_at TIMESTAMP); @@ -156,7 +147,7 @@ def _json_path_escape(key: str) -> str: _INSERTS = { "questions": "INSERT INTO questions VALUES (?, ?, ?, ?, ?)", - "question_columns": "INSERT INTO question_columns VALUES (?, ?, ?)", + "question_columns": "INSERT INTO question_columns VALUES (?, ?)", "records": "INSERT INTO records VALUES (?, ?, ?, ?)", "suggestions": "INSERT INTO suggestions VALUES (?, ?, ?, ?, ?)", "responses": "INSERT INTO responses VALUES (?, ?, ?)", @@ -240,18 +231,23 @@ def _json_path_escape(key: str) -> str: table_object_rows AS ( SELECT * FROM table_rows WHERE json_type(row_json) = 'OBJECT' ), -table_cell_values AS ( - -- the pre-escaped `sub_path` quoted into the JSON path handles arbitrary key names - SELECT tr.reference, tr.record_id, tr.row_idx, - tr.schema_name || '.' || tr.question_name || '.' || qc.sub_column AS column_name, - json_extract(tr.row_json, '$."' || qc.sub_path || '"') AS value, - tr.source, tr.agent, tr.score - FROM table_object_rows tr - JOIN question_columns qc ON qc.question_id = tr.question_id +table_row_entries AS ( + -- same zip-unnest as the response envelope: no data-derived text ever reaches a JSON path, + -- so quotes, backslashes, empty names and '*' in a binding are all just ordinary keys + SELECT reference, record_id, question_id, schema_name, question_name, source, agent, score, row_idx, + unnest(json_keys(row_json)) AS entry_key, + unnest(json_extract(row_json, '$.*')) AS entry_value + FROM table_object_rows ), table_cells AS ( - -- absent sub-key (SQL NULL) and JSON-null alike are omitted - SELECT * FROM table_cell_values WHERE json_type(value) <> 'NULL' + -- an unmatched binding is an absent sub-key: no join row, hence no cell. JSON-null omitted too. + SELECT e.reference, e.record_id, e.row_idx, + e.schema_name || '.' || e.question_name || '.' || qc.sub_column AS column_name, + e.entry_value AS value, + e.source, e.agent, e.score + FROM table_row_entries e + JOIN question_columns qc ON qc.question_id = e.question_id AND qc.sub_column = e.entry_key + WHERE json_type(e.entry_value) <> 'NULL' ), fanout AS ( SELECT reference, max(row_idx) AS max_idx FROM table_object_rows GROUP BY reference @@ -389,10 +385,7 @@ async def build_workspace_view(db: AsyncSession, *, workspace_id: UUID, offset: (str(q.id), str(q.schema_id), schema_names[q.schema_id], q.name, q.type.value) for q in questions ], "question_columns": [ - (str(q.id), sub, _json_path_escape(sub)) - for q in questions - if q.type == QuestionType.table - for sub in (q.columns or []) + (str(q.id), sub) for q in questions if q.type == QuestionType.table for sub in (q.columns or []) ], "records": [(str(r.id), str(r.schema_id), r.reference, r.inserted_at) for r in records], "suggestions": [ diff --git a/extralit-server/tests/integration/contexts/v2/test_workspace_projection.py b/extralit-server/tests/integration/contexts/v2/test_workspace_projection.py index e2e2a8ee0..fff5af555 100644 --- a/extralit-server/tests/integration/contexts/v2/test_workspace_projection.py +++ b/extralit-server/tests/integration/contexts/v2/test_workspace_projection.py @@ -154,15 +154,20 @@ async def test_single_dict_table_value_is_one_row(db): assert view.rows[0].cells["Outcomes.results.value"].value == "12%" -async def test_quotes_and_backslashes_in_names_do_not_break_json_paths(db): - """Question names and sub-column bindings are unconstrained user input; a raw `"` used to be - a bind-time JSON-path error that failed the whole grid, not just the offending cell.""" +async def test_hostile_names_are_treated_as_ordinary_keys(db): + """Question names and sub-column bindings are unconstrained user input. Interpolating them + into a JSON path made a quote, a backslash or an empty name abort the WHOLE grid, and made a + binding named `*` silently return every key's value.""" workspace = await WorkspaceFactory.create() schema, version = await _make_schema(workspace, "Outcomes") await _add_question(schema, 'we"ird\\name') - t = await _add_question(schema, "results", qtype=QuestionType.table, columns=['sub"col', "back\\slash"]) + t = await _add_question(schema, "results", qtype=QuestionType.table, columns=['sub"col', "back\\slash", "", "*"]) rec = await V2RecordFactory.create(version=version, reference="10.1/a") - await V2SuggestionFactory.create(record=rec, question=t, value=[{'sub"col': "A", "back\\slash": "B"}]) + await V2SuggestionFactory.create( + record=rec, + question=t, + value=[{'sub"col': "A", "back\\slash": "B", "": "C", "*": "D", "unbound": "E"}], + ) user = await UserFactory.create() await V2ResponseFactory.create( record=rec, user=user, values={'we"ird\\name': {"value": "RCT"}}, status=ResponseStatus.submitted @@ -174,12 +179,17 @@ async def test_quotes_and_backslashes_in_names_do_not_break_json_paths(db): 'Outcomes.we"ird\\name', 'Outcomes.results.sub"col', "Outcomes.results.back\\slash", + "Outcomes.results.", + "Outcomes.results.*", ] cells = view.rows[0].cells assert cells['Outcomes.we"ird\\name'].value == "RCT" assert cells['Outcomes.we"ird\\name'].source == "response" assert cells['Outcomes.results.sub"col'].value == "A" assert cells["Outcomes.results.back\\slash"].value == "B" + assert cells["Outcomes.results."].value == "C" + assert cells["Outcomes.results.*"].value == "D" # a literal key, not a wildcard + assert "Outcomes.results.unbound" not in cells async def test_effective_record_is_latest_inserted_per_reference_schema(db): From d9a12e018230d1041c40dad912d4e47b28876ae8 Mon Sep 17 00:00:00 2001 From: JonnyTran Date: Tue, 21 Jul 2026 11:38:08 -0700 Subject: [PATCH 10/21] feat(server): GET /api/v2/projection workspace endpoint --- .../src/extralit_server/api/v2/projection.py | 15 +++++- .../integration/api/v2/test_projection.py | 51 +++++++++++++++++++ 2 files changed, 65 insertions(+), 1 deletion(-) diff --git a/extralit-server/src/extralit_server/api/v2/projection.py b/extralit-server/src/extralit_server/api/v2/projection.py index da1ba1223..de60d6694 100644 --- a/extralit-server/src/extralit_server/api/v2/projection.py +++ b/extralit-server/src/extralit_server/api/v2/projection.py @@ -5,7 +5,7 @@ from sqlalchemy.ext.asyncio import AsyncSession from extralit_server.api.policies.v1 import SchemaPolicy, authorize -from extralit_server.api.schemas.v2.projection import ProjectionView +from extralit_server.api.schemas.v2.projection import ProjectionView, WorkspaceProjection from extralit_server.contexts.v2 import projection as projection_ctx from extralit_server.database import get_async_db from extralit_server.models import User @@ -14,6 +14,19 @@ router = APIRouter(tags=["v2: projection"]) +@router.get("/projection", response_model=WorkspaceProjection) +async def get_workspace_projection( + *, + db: Annotated[AsyncSession, Depends(get_async_db)], + current_user: Annotated[User, Security(auth.get_current_user)], + workspace_id: Annotated[UUID, Query(description="Workspace to scope the view (required)")], + offset: Annotated[int, Query(ge=0, description="Reference offset (not fan-out rows)")] = 0, + limit: Annotated[int, Query(ge=1, le=100, description="References per page")] = 50, +): + await authorize(current_user, SchemaPolicy.list(workspace_id)) + return await projection_ctx.build_workspace_view(db, workspace_id=workspace_id, offset=offset, limit=limit) + + # Distinct `/projection/...` prefix, NOT `/references/{reference:path}/view`: the greedy `:path` # converter on the existing GET /references/{reference:path} (Phase 3) would otherwise shadow a # `/view` suffix, and a real reference ending in "/view" would collide. See spec §17.4. diff --git a/extralit-server/tests/integration/api/v2/test_projection.py b/extralit-server/tests/integration/api/v2/test_projection.py index e1e439cf8..0526a5ad0 100644 --- a/extralit-server/tests/integration/api/v2/test_projection.py +++ b/extralit-server/tests/integration/api/v2/test_projection.py @@ -59,3 +59,54 @@ async def test_projection_view_unknown_reference_returns_empty(async_client, own assert body["reference"] == "doc-nope" assert body["total_records"] == 0 assert body["records"] == [] + + +async def test_workspace_projection_returns_manifest_rows_and_total(async_client, owner_auth_header): + workspace = await WorkspaceFactory.create() + schema, version, q = await _schema_with_question(workspace) + record = await V2RecordFactory.create(version=version, reference="doc-1") + await V2SuggestionFactory.create(record=record, question=q, value="flu", agent="gpt-x", score=0.92) + + resp = await async_client.get(f"/api/v2/projection?workspace_id={workspace.id}", headers=owner_auth_header) + + assert resp.status_code == 200, resp.text + body = resp.json() + assert body["total_references"] == 1 + [column] = [c for c in body["columns"] if c["question_name"] == q.name] + assert column["schema_id"] == str(schema.id) + assert column["sub_column"] is None + [row] = body["rows"] + assert row["reference"] == "doc-1" + assert row["row_index"] == 0 + cell = row["cells"][column["name"]] + assert cell == { + "value": "flu", + "source": "suggestion", + "record_id": str(record.id), + "agent": "gpt-x", + "score": 0.92, + } + + +async def test_workspace_projection_paginates_references(async_client, owner_auth_header): + workspace = await WorkspaceFactory.create() + _schema, version, _q = await _schema_with_question(workspace) + for i in range(3): + await V2RecordFactory.create(version=version, reference=f"doc-{i}") + + resp = await async_client.get( + f"/api/v2/projection?workspace_id={workspace.id}&offset=1&limit=1", headers=owner_auth_header + ) + + assert resp.status_code == 200, resp.text + body = resp.json() + assert body["total_references"] == 3 + assert [r["reference"] for r in body["rows"]] == ["doc-1"] + + +async def test_workspace_projection_rejects_limit_over_100(async_client, owner_auth_header): + workspace = await WorkspaceFactory.create() + resp = await async_client.get( + f"/api/v2/projection?workspace_id={workspace.id}&limit=101", headers=owner_auth_header + ) + assert resp.status_code == 422 From 6f013f904feb63ea45ce8e1262c6c9622805cae5 Mon Sep 17 00:00:00 2001 From: JonnyTran Date: Tue, 21 Jul 2026 11:45:15 -0700 Subject: [PATCH 11/21] test(server): pin projection authz and query-bounds validation --- .../integration/api/v2/test_projection.py | 40 +++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/extralit-server/tests/integration/api/v2/test_projection.py b/extralit-server/tests/integration/api/v2/test_projection.py index 0526a5ad0..453376075 100644 --- a/extralit-server/tests/integration/api/v2/test_projection.py +++ b/extralit-server/tests/integration/api/v2/test_projection.py @@ -8,6 +8,7 @@ V2RecordFactory, V2SuggestionFactory, WorkspaceFactory, + WorkspaceUserFactory, ) pytestmark = pytest.mark.asyncio @@ -110,3 +111,42 @@ async def test_workspace_projection_rejects_limit_over_100(async_client, owner_a f"/api/v2/projection?workspace_id={workspace.id}&limit=101", headers=owner_auth_header ) assert resp.status_code == 422 + + +async def test_workspace_projection_authz(async_client, annotator_auth_header, annotator): + workspace = await WorkspaceFactory.create() + _schema, version, q = await _schema_with_question(workspace) + record = await V2RecordFactory.create(version=version, reference="doc-1") + await V2SuggestionFactory.create(record=record, question=q, value="flu", agent="gpt-x", score=0.92) + + # Non-member: forbidden. + resp = await async_client.get(f"/api/v2/projection?workspace_id={workspace.id}", headers=annotator_auth_header) + assert resp.status_code == 403, resp.text + + # Member annotator: allowed to read. + await WorkspaceUserFactory.create(workspace_id=workspace.id, user_id=annotator.id) + resp = await async_client.get(f"/api/v2/projection?workspace_id={workspace.id}", headers=annotator_auth_header) + assert resp.status_code == 200, resp.text + assert resp.json()["total_references"] == 1 + + +@pytest.mark.parametrize( + ("query_params", "needs_workspace"), + [ + ("limit=0", True), + ("offset=-1", True), + ("workspace_id=not-a-uuid", False), + ("", False), # workspace_id missing entirely + ], + ids=["limit_below_minimum", "offset_negative", "workspace_id_malformed", "workspace_id_missing"], +) +async def test_workspace_projection_rejects_invalid_query_params( + async_client, owner_auth_header, query_params, needs_workspace +): + query = query_params + if needs_workspace: + workspace = await WorkspaceFactory.create() + query = f"workspace_id={workspace.id}&{query_params}" + + resp = await async_client.get(f"/api/v2/projection?{query}", headers=owner_auth_header) + assert resp.status_code == 422, resp.text From 7bb8e2351391aeb36de94788c759c658773637d0 Mon Sep 17 00:00:00 2001 From: JonnyTran Date: Tue, 21 Jul 2026 11:49:56 -0700 Subject: [PATCH 12/21] feat(v2-ui): workspace projection contract, domain types, repository method --- .../projection/WorkspaceProjection.ts | 30 ++ .../v2/infrastructure/api/generated/v2-api.ts | 218 +++++++++++ .../v2/infrastructure/api/openapi.json | 357 ++++++++++++++++++ .../repositories/ProjectionRepository.test.ts | 60 +++ .../repositories/ProjectionRepository.ts | 42 +++ 5 files changed, 707 insertions(+) create mode 100644 extralit-frontend/v2/domain/entities/projection/WorkspaceProjection.ts diff --git a/extralit-frontend/v2/domain/entities/projection/WorkspaceProjection.ts b/extralit-frontend/v2/domain/entities/projection/WorkspaceProjection.ts new file mode 100644 index 000000000..cc5cb1853 --- /dev/null +++ b/extralit-frontend/v2/domain/entities/projection/WorkspaceProjection.ts @@ -0,0 +1,30 @@ +export interface ProjectionColumn { + name: string; // flat "Schema.question" / "Schema.question.subcol" + schemaId: string; + schemaName: string; + questionName: string; + subColumn: string | null; + dtype: string; +} + +export interface ProjectionGridCell { + value: unknown; + source: "response" | "suggestion"; + recordId: string; + agent: string | null; + score: number | number[] | null; +} + +export interface ProjectionGridRow { + reference: string; + rowIndex: number; + cells: Record; +} + +export class WorkspaceProjection { + constructor( + public readonly columns: ProjectionColumn[], + public readonly rows: ProjectionGridRow[], + public readonly totalReferences: number + ) {} +} diff --git a/extralit-frontend/v2/infrastructure/api/generated/v2-api.ts b/extralit-frontend/v2/infrastructure/api/generated/v2-api.ts index 99da1305a..ca9a055a6 100644 --- a/extralit-frontend/v2/infrastructure/api/generated/v2-api.ts +++ b/extralit-frontend/v2/infrastructure/api/generated/v2-api.ts @@ -4,6 +4,23 @@ */ export interface paths { + "/projection": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + /** Get Workspace Projection */ + get: operations["get_workspace_projection_projection_get"]; + put?: never; + post?: never; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; "/projection/references/{reference}": { parameters: { query?: never; @@ -341,8 +358,14 @@ export interface components { }; /** ProjectionCell */ ProjectionCell: { + /** Agent */ + agent?: string | null; /** Question Name */ question_name: string; + /** Record Id */ + record_id?: string | null; + /** Score */ + score?: number | number[] | null; /** Source */ source?: ("response" | "suggestion") | null; /** Value */ @@ -816,6 +839,63 @@ export interface components { * @enum {string} */ V2RecordStatus: "pending" | "completed" | "discarded"; + /** WorkspaceProjection */ + WorkspaceProjection: { + /** Columns */ + columns: components["schemas"]["WorkspaceProjectionColumn"][]; + /** Rows */ + rows: components["schemas"]["WorkspaceProjectionRow"][]; + /** Total References */ + total_references: number; + }; + /** WorkspaceProjectionCell */ + WorkspaceProjectionCell: { + /** Agent */ + agent?: string | null; + /** + * Record Id + * Format: uuid + */ + record_id: string; + /** Score */ + score?: number | number[] | null; + /** + * Source + * @enum {string} + */ + source: "response" | "suggestion"; + /** Value */ + value?: unknown | null; + }; + /** WorkspaceProjectionColumn */ + WorkspaceProjectionColumn: { + /** Dtype */ + dtype: string; + /** Name */ + name: string; + /** Question Name */ + question_name: string; + /** + * Schema Id + * Format: uuid + */ + schema_id: string; + /** Schema Name */ + schema_name: string; + /** Sub Column */ + sub_column?: string | null; + }; + /** WorkspaceProjectionRow */ + WorkspaceProjectionRow: { + /** Cells */ + cells: { + [key: string]: components["schemas"]["WorkspaceProjectionCell"]; + }; + /** Reference */ + reference: string; + /** Row Index */ + row_index: number; + }; }; responses: never; parameters: never; @@ -825,6 +905,144 @@ export interface components { } export type $defs = Record; export interface operations { + get_workspace_projection_projection_get: { + parameters: { + query: { + /** @description Workspace to scope the view (required) */ + workspace_id: string; + /** @description Reference offset (not fan-out rows) */ + offset?: number; + /** @description References per page */ + limit?: number; + }; + header?: never; + path?: never; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description Successful Response */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["WorkspaceProjection"]; + }; + }; + /** @description Bad Request */ + 400: { + headers: { + [name: string]: unknown; + }; + content: { + /** + * @example { + * "detail": { + * "code": "extralit.api.errors::MissingDatasetRecordsError", + * "params": { + * "extra": "error parameters" + * } + * } + * } + */ + "application/json": unknown; + }; + }; + /** @description Forbidden */ + 403: { + headers: { + [name: string]: unknown; + }; + content: { + /** + * @example { + * "detail": { + * "code": "extralit.api.errors::ForbiddenOperationError", + * "params": { + * "extra": "error parameters" + * } + * } + * } + */ + "application/json": unknown; + }; + }; + /** @description Not Found */ + 404: { + headers: { + [name: string]: unknown; + }; + content: { + /** + * @example { + * "detail": { + * "code": "extralit.api.errors::EntityNotFoundError", + * "params": { + * "extra": "error parameters" + * } + * } + * } + */ + "application/json": unknown; + }; + }; + /** @description Conflict */ + 409: { + headers: { + [name: string]: unknown; + }; + content: { + /** + * @example { + * "detail": { + * "code": "extralit.api.errors::EntityAlreadyExistsError", + * "params": { + * "extra": "error parameters" + * } + * } + * } + */ + "application/json": unknown; + }; + }; + /** @description Unprocessable Content */ + 422: { + headers: { + [name: string]: unknown; + }; + content: { + /** + * @example { + * "detail": { + * "code": "extralit.api.errors::ValidationError", + * "params": { + * "extra": "error parameters" + * } + * } + * } + */ + "application/json": unknown; + }; + }; + /** @description Internal Server Error */ + 500: { + headers: { + [name: string]: unknown; + }; + content: { + /** + * @example { + * "detail": { + * "code": "builtins.TypeError" + * } + * } + */ + "application/json": unknown; + }; + }; + }; + }; get_reference_projection_projection_references__reference__get: { parameters: { query: { diff --git a/extralit-frontend/v2/infrastructure/api/openapi.json b/extralit-frontend/v2/infrastructure/api/openapi.json index bb27a0dd0..e75ed373e 100644 --- a/extralit-frontend/v2/infrastructure/api/openapi.json +++ b/extralit-frontend/v2/infrastructure/api/openapi.json @@ -21,10 +21,50 @@ }, "ProjectionCell": { "properties": { + "agent": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Agent" + }, "question_name": { "title": "Question Name", "type": "string" }, + "record_id": { + "anyOf": [ + { + "format": "uuid", + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Record Id" + }, + "score": { + "anyOf": [ + { + "type": "number" + }, + { + "items": { + "type": "number" + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "Score" + }, "source": { "anyOf": [ { @@ -1188,6 +1228,166 @@ ], "title": "V2RecordStatus", "type": "string" + }, + "WorkspaceProjection": { + "properties": { + "columns": { + "items": { + "$ref": "#/components/schemas/WorkspaceProjectionColumn" + }, + "title": "Columns", + "type": "array" + }, + "rows": { + "items": { + "$ref": "#/components/schemas/WorkspaceProjectionRow" + }, + "title": "Rows", + "type": "array" + }, + "total_references": { + "title": "Total References", + "type": "integer" + } + }, + "required": [ + "columns", + "rows", + "total_references" + ], + "title": "WorkspaceProjection", + "type": "object" + }, + "WorkspaceProjectionCell": { + "properties": { + "agent": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Agent" + }, + "record_id": { + "format": "uuid", + "title": "Record Id", + "type": "string" + }, + "score": { + "anyOf": [ + { + "type": "number" + }, + { + "items": { + "type": "number" + }, + "type": "array" + }, + { + "type": "null" + } + ], + "title": "Score" + }, + "source": { + "enum": [ + "response", + "suggestion" + ], + "title": "Source", + "type": "string" + }, + "value": { + "anyOf": [ + {}, + { + "type": "null" + } + ], + "title": "Value" + } + }, + "required": [ + "source", + "record_id" + ], + "title": "WorkspaceProjectionCell", + "type": "object" + }, + "WorkspaceProjectionColumn": { + "properties": { + "dtype": { + "title": "Dtype", + "type": "string" + }, + "name": { + "title": "Name", + "type": "string" + }, + "question_name": { + "title": "Question Name", + "type": "string" + }, + "schema_id": { + "format": "uuid", + "title": "Schema Id", + "type": "string" + }, + "schema_name": { + "title": "Schema Name", + "type": "string" + }, + "sub_column": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Sub Column" + } + }, + "required": [ + "name", + "schema_id", + "schema_name", + "question_name", + "dtype" + ], + "title": "WorkspaceProjectionColumn", + "type": "object" + }, + "WorkspaceProjectionRow": { + "properties": { + "cells": { + "additionalProperties": { + "$ref": "#/components/schemas/WorkspaceProjectionCell" + }, + "title": "Cells", + "type": "object" + }, + "reference": { + "title": "Reference", + "type": "string" + }, + "row_index": { + "title": "Row Index", + "type": "integer" + } + }, + "required": [ + "reference", + "row_index", + "cells" + ], + "title": "WorkspaceProjectionRow", + "type": "object" } }, "securitySchemes": { @@ -1209,6 +1409,163 @@ }, "openapi": "3.1.0", "paths": { + "/projection": { + "get": { + "operationId": "get_workspace_projection_projection_get", + "parameters": [ + { + "description": "Workspace to scope the view (required)", + "in": "query", + "name": "workspace_id", + "required": true, + "schema": { + "description": "Workspace to scope the view (required)", + "format": "uuid", + "title": "Workspace Id", + "type": "string" + } + }, + { + "description": "Reference offset (not fan-out rows)", + "in": "query", + "name": "offset", + "required": false, + "schema": { + "default": 0, + "description": "Reference offset (not fan-out rows)", + "minimum": 0, + "title": "Offset", + "type": "integer" + } + }, + { + "description": "References per page", + "in": "query", + "name": "limit", + "required": false, + "schema": { + "default": 50, + "description": "References per page", + "maximum": 100, + "minimum": 1, + "title": "Limit", + "type": "integer" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/WorkspaceProjection" + } + } + }, + "description": "Successful Response" + }, + "400": { + "content": { + "application/json": { + "example": { + "detail": { + "code": "extralit.api.errors::MissingDatasetRecordsError", + "params": { + "extra": "error parameters" + } + } + } + } + }, + "description": "Bad Request" + }, + "403": { + "content": { + "application/json": { + "example": { + "detail": { + "code": "extralit.api.errors::ForbiddenOperationError", + "params": { + "extra": "error parameters" + } + } + } + } + }, + "description": "Forbidden" + }, + "404": { + "content": { + "application/json": { + "example": { + "detail": { + "code": "extralit.api.errors::EntityNotFoundError", + "params": { + "extra": "error parameters" + } + } + } + } + }, + "description": "Not Found" + }, + "409": { + "content": { + "application/json": { + "example": { + "detail": { + "code": "extralit.api.errors::EntityAlreadyExistsError", + "params": { + "extra": "error parameters" + } + } + } + } + }, + "description": "Conflict" + }, + "422": { + "content": { + "application/json": { + "example": { + "detail": { + "code": "extralit.api.errors::ValidationError", + "params": { + "extra": "error parameters" + } + } + } + } + }, + "description": "Unprocessable Content" + }, + "500": { + "content": { + "application/json": { + "example": { + "detail": { + "code": "builtins.TypeError" + } + } + } + }, + "description": "Internal Server Error" + } + }, + "security": [ + { + "APIKeyHeader": [] + }, + { + "HTTPBearer": [] + } + ], + "summary": "Get Workspace Projection", + "tags": [ + "v2: projection" + ] + } + }, "/projection/references/{reference}": { "get": { "operationId": "get_reference_projection_projection_references__reference__get", diff --git a/extralit-frontend/v2/infrastructure/repositories/ProjectionRepository.test.ts b/extralit-frontend/v2/infrastructure/repositories/ProjectionRepository.test.ts index 402eddbf0..be2e83ab8 100644 --- a/extralit-frontend/v2/infrastructure/repositories/ProjectionRepository.test.ts +++ b/extralit-frontend/v2/infrastructure/repositories/ProjectionRepository.test.ts @@ -44,3 +44,63 @@ describe("ProjectionRepository", () => { }); }); }); + +const BACKEND_WORKSPACE = { + columns: [ + { + name: "Design.type", + schema_id: "s-1", + schema_name: "Design", + question_name: "type", + sub_column: null, + dtype: "text", + }, + ], + rows: [ + { + reference: "10.1000/j.x", + row_index: 0, + cells: { + "Design.type": { value: "RCT", source: "response", record_id: "r-1", agent: null, score: null }, + }, + }, + ], + total_references: 213, +}; + +describe("getWorkspaceProjection", () => { + it("pages the workspace projection and maps snake_case to the domain shape", async () => { + const axios = { get: vi.fn(async () => ({ data: BACKEND_WORKSPACE })) }; + const page = await new ProjectionRepository(axios as never).getWorkspaceProjection("w-1", 50, 25); + + expect(axios.get).toHaveBeenCalledWith("/v2/projection", { + params: { workspace_id: "w-1", offset: 50, limit: 25 }, + }); + expect(page.totalReferences).toBe(213); + expect(page.columns[0]).toEqual({ + name: "Design.type", + schemaId: "s-1", + schemaName: "Design", + questionName: "type", + subColumn: null, + dtype: "text", + }); + expect(page.rows[0].reference).toBe("10.1000/j.x"); + expect(page.rows[0].rowIndex).toBe(0); + expect(page.rows[0].cells["Design.type"]).toEqual({ + value: "RCT", + source: "response", + recordId: "r-1", + agent: null, + score: null, + }); + }); + + it("defaults to offset 0, limit 50", async () => { + const axios = { get: vi.fn(async () => ({ data: BACKEND_WORKSPACE })) }; + await new ProjectionRepository(axios as never).getWorkspaceProjection("w-1"); + expect(axios.get).toHaveBeenCalledWith("/v2/projection", { + params: { workspace_id: "w-1", offset: 0, limit: 50 }, + }); + }); +}); diff --git a/extralit-frontend/v2/infrastructure/repositories/ProjectionRepository.ts b/extralit-frontend/v2/infrastructure/repositories/ProjectionRepository.ts index 1d9808983..0d017ea6e 100644 --- a/extralit-frontend/v2/infrastructure/repositories/ProjectionRepository.ts +++ b/extralit-frontend/v2/infrastructure/repositories/ProjectionRepository.ts @@ -1,7 +1,9 @@ import type { AxiosInstance } from "axios"; import type { components } from "../api/generated/v2-api"; +import { type ProjectionColumn, type ProjectionGridRow } from "../../domain/entities/projection/WorkspaceProjection"; type BackendProjectionView = components["schemas"]["ProjectionView"]; +type BackendWorkspaceProjection = components["schemas"]["WorkspaceProjection"]; export interface ProjectionCellDto { questionName: string; @@ -22,6 +24,12 @@ export interface ProjectionViewDto { totalRecords: number; } +export interface WorkspaceProjectionPageDto { + columns: ProjectionColumn[]; + rows: ProjectionGridRow[]; + totalReferences: number; +} + export class ProjectionRepository { constructor(private readonly axios: AxiosInstance) {} @@ -46,4 +54,38 @@ export class ProjectionRepository { })), }; } + + async getWorkspaceProjection(workspaceId: string, offset = 0, limit = 50): Promise { + const { data } = await this.axios.get("/v2/projection", { + params: { workspace_id: workspaceId, offset, limit }, + }); + return { + // Column order is server-defined (schema definition order, not alphabetical) — preserve as-is. + columns: data.columns.map((c) => ({ + name: c.name, + schemaId: c.schema_id, + schemaName: c.schema_name, + questionName: c.question_name, + subColumn: c.sub_column ?? null, + dtype: c.dtype, + })), + rows: data.rows.map((r) => ({ + reference: r.reference, + rowIndex: r.row_index, + cells: Object.fromEntries( + Object.entries(r.cells).map(([columnName, cell]) => [ + columnName, + { + value: cell.value ?? null, + source: cell.source, + recordId: cell.record_id, + agent: cell.agent ?? null, + score: cell.score ?? null, + }, + ]) + ), + })), + totalReferences: data.total_references, + }; + } } From 6d56edab4e84d0be945e3dba369dbe8f62d08a58 Mon Sep 17 00:00:00 2001 From: JonnyTran Date: Tue, 21 Jul 2026 11:57:11 -0700 Subject: [PATCH 13/21] =?UTF-8?q?feat(v2-ui):=20pure=20grid=20adapter=20?= =?UTF-8?q?=E2=80=94=20flat=20rows,=20banding,=20click-URL=20contract?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../entities/projection/grid-adapter.test.ts | 77 ++++++++++++++++++ .../entities/projection/grid-adapter.ts | 78 +++++++++++++++++++ 2 files changed, 155 insertions(+) create mode 100644 extralit-frontend/v2/domain/entities/projection/grid-adapter.test.ts create mode 100644 extralit-frontend/v2/domain/entities/projection/grid-adapter.ts diff --git a/extralit-frontend/v2/domain/entities/projection/grid-adapter.test.ts b/extralit-frontend/v2/domain/entities/projection/grid-adapter.test.ts new file mode 100644 index 000000000..715000628 --- /dev/null +++ b/extralit-frontend/v2/domain/entities/projection/grid-adapter.test.ts @@ -0,0 +1,77 @@ +import { describe, expect, it } from "vitest"; +import { + ANNOTATION_CELL_LINKS_ENABLED, + bandParity, + buildAnnotationUrl, + cellAt, + REFERENCE_COLUMN, + toPerspectiveData, +} from "./grid-adapter"; +import { WorkspaceProjection, type ProjectionGridCell } from "./WorkspaceProjection"; + +const cell = (value: unknown): ProjectionGridCell => ({ + value, + source: "suggestion", + recordId: "r-1", + agent: "gpt-x", + score: 0.9, +}); + +const COLUMNS = [ + { name: "Design.type", schemaId: "s-1", schemaName: "Design", questionName: "type", subColumn: null, dtype: "text" }, + { + name: "Outcomes.results.value", + schemaId: "s-2", + schemaName: "Outcomes", + questionName: "results", + subColumn: "value", + dtype: "table", + }, +]; + +const PROJECTION = new WorkspaceProjection( + COLUMNS, + [ + { reference: "10.1/a", rowIndex: 0, cells: { "Design.type": cell("RCT") } }, + { reference: "10.1/a", rowIndex: 1, cells: { "Outcomes.results.value": cell("12%") } }, + { reference: "10.1/b", rowIndex: 0, cells: {} }, + ], + 2 +); + +describe("toPerspectiveData", () => { + it("emits one flat record per row with every manifest column (null when absent)", () => { + expect(toPerspectiveData(PROJECTION)).toEqual([ + { [REFERENCE_COLUMN]: "10.1/a", "Design.type": "RCT", "Outcomes.results.value": null }, + { [REFERENCE_COLUMN]: "10.1/a", "Design.type": null, "Outcomes.results.value": "12%" }, + { [REFERENCE_COLUMN]: "10.1/b", "Design.type": null, "Outcomes.results.value": null }, + ]); + }); +}); + +describe("cellAt", () => { + it("returns the enriched cell for the loaded row order", () => { + expect(cellAt(PROJECTION, 0, "Design.type")?.recordId).toBe("r-1"); + }); + + it("returns null for empty cells and out-of-range rows", () => { + expect(cellAt(PROJECTION, 2, "Design.type")).toBeNull(); + expect(cellAt(PROJECTION, 99, "Design.type")).toBeNull(); + }); +}); + +describe("bandParity", () => { + it("flips parity when the reference changes, not per row", () => { + expect(bandParity(PROJECTION)).toEqual([0, 0, 1]); + }); +}); + +describe("annotation URL contract (spec §3.3)", () => { + it("stays guarded off in this build", () => { + expect(ANNOTATION_CELL_LINKS_ENABLED).toBe(false); + }); + + it("puts the schema id in the dataset slot and encodes the reference", () => { + expect(buildAnnotationUrl("s-1", "10.1/a b")).toBe("/dataset/s-1/annotation-mode?_search=10.1%2Fa%20b"); + }); +}); diff --git a/extralit-frontend/v2/domain/entities/projection/grid-adapter.ts b/extralit-frontend/v2/domain/entities/projection/grid-adapter.ts new file mode 100644 index 000000000..7a11aefb5 --- /dev/null +++ b/extralit-frontend/v2/domain/entities/projection/grid-adapter.ts @@ -0,0 +1,78 @@ +import { type WorkspaceProjection, type ProjectionGridCell } from "./WorkspaceProjection"; + +/** + * Flat-row column name carrying the reference (DOI/identifier) for each row. + * Kept out of `projection.columns` since it is a manifest-independent, always-present key. + */ +export const REFERENCE_COLUMN = "reference"; + +/** + * Maps a `WorkspaceProjection` into the flat-row shape Perspective ingests. + * + * `projection.columns` order is the single source of truth for column order (server-side + * question order is definition order, not alphabetical, and JS object key order is not a + * safe substitute once column names could look numeric). `reference` is emitted first so + * every row shares an identical, stably-ordered key set — every manifest column is present + * on every row (`null` when the cell is absent) so Perspective infers one stable schema + * instead of a per-row-varying one. + */ +export function toPerspectiveData(projection: WorkspaceProjection): Record[] { + return projection.rows.map((row) => { + const record: Record = { [REFERENCE_COLUMN]: row.reference }; + for (const column of projection.columns) { + const cell = row.cells[column.name]; + record[column.name] = cell ? cell.value : null; + } + return record; + }); +} + +/** + * Looks up the enriched cell (value + provenance) for a grid position. + * + * Relies on the static-grid invariant: the projection is loaded once, unsorted and + * unfiltered, so a datagrid row index maps 1:1 onto `projection.rows` by array position. + * Returns null for an absent cell or an out-of-range row index. + */ +export function cellAt( + projection: WorkspaceProjection, + rowIndex: number, + columnName: string +): ProjectionGridCell | null { + const row = projection.rows[rowIndex]; + if (!row) { + return null; + } + return row.cells[columnName] ?? null; +} + +/** + * Returns a 0/1 parity value per row that flips whenever `reference` changes from the + * previous row (not per row). Perspective has no notion of merged/grouped cells, so this + * alternating band is the §3.1 reference-grouping affordance rendered as a background tint. + */ +export function bandParity(projection: WorkspaceProjection): number[] { + let parity = 0; + let previousReference: string | null = null; + return projection.rows.map((row) => { + if (previousReference !== null && row.reference !== previousReference) { + parity = parity === 0 ? 1 : 0; + } + previousReference = row.reference; + return parity; + }); +} + +/** + * Guards the §3.3 click-to-annotate affordance off until annotation-mode resolves v2 schema + * ids (see ledger §5). Flip to true once that lands. + */ +export const ANNOTATION_CELL_LINKS_ENABLED = false; + +/** + * Builds the annotation-mode deep link for a cell's reference, percent-encoding the + * reference so slashes and other reserved characters survive the query string. + */ +export function buildAnnotationUrl(schemaId: string, reference: string): string { + return `/dataset/${schemaId}/annotation-mode?_search=${encodeURIComponent(reference)}`; +} From 9af4e71b41a0767509610150486720e39329ab8e Mon Sep 17 00:00:00 2001 From: JonnyTran Date: Tue, 21 Jul 2026 12:01:00 -0700 Subject: [PATCH 14/21] test(v2-ui): pin server column/cell ordering in getWorkspaceProjection --- .../repositories/ProjectionRepository.test.ts | 125 ++++++++++-------- 1 file changed, 72 insertions(+), 53 deletions(-) diff --git a/extralit-frontend/v2/infrastructure/repositories/ProjectionRepository.test.ts b/extralit-frontend/v2/infrastructure/repositories/ProjectionRepository.test.ts index be2e83ab8..9ad9dcbd7 100644 --- a/extralit-frontend/v2/infrastructure/repositories/ProjectionRepository.test.ts +++ b/extralit-frontend/v2/infrastructure/repositories/ProjectionRepository.test.ts @@ -18,6 +18,41 @@ const BACKEND_VIEW = { ], }; +// Columns deliberately out of alphabetical order — the server orders columns by schema/question +// definition order, not alphabetically, and nothing on the server pins that order. The grid's +// column layout depends on it, so the mapping must preserve exactly what the server sent. +const BACKEND_WORKSPACE = { + columns: [ + { + name: "Zeta.x", + schema_id: "s-2", + schema_name: "Zeta", + question_name: "x", + sub_column: null, + dtype: "text", + }, + { + name: "Alpha.y", + schema_id: "s-1", + schema_name: "Alpha", + question_name: "y", + sub_column: null, + dtype: "text", + }, + ], + rows: [ + { + reference: "10.1000/j.x", + row_index: 0, + cells: { + "Zeta.x": { value: "RCT", source: "response", record_id: "r-1", agent: null, score: null }, + "Alpha.y": { value: "42", source: "response", record_id: "r-2", agent: null, score: null }, + }, + }, + ], + total_references: 213, +}; + describe("ProjectionRepository", () => { it("percent-encodes the slashed reference and maps the view to DTOs (seam B)", async () => { const axios = { get: vi.fn(async () => ({ data: BACKEND_VIEW })) } as unknown as AxiosInstance; @@ -43,64 +78,48 @@ describe("ProjectionRepository", () => { ], }); }); -}); -const BACKEND_WORKSPACE = { - columns: [ - { - name: "Design.type", - schema_id: "s-1", - schema_name: "Design", - question_name: "type", - sub_column: null, - dtype: "text", - }, - ], - rows: [ - { - reference: "10.1000/j.x", - row_index: 0, - cells: { - "Design.type": { value: "RCT", source: "response", record_id: "r-1", agent: null, score: null }, - }, - }, - ], - total_references: 213, -}; + describe("getWorkspaceProjection", () => { + it("pages the workspace projection and maps snake_case to the domain shape, preserving server column/cell order", async () => { + const axios = { get: vi.fn(async () => ({ data: BACKEND_WORKSPACE })) }; + const page = await new ProjectionRepository(axios as never).getWorkspaceProjection("w-1", 50, 25); -describe("getWorkspaceProjection", () => { - it("pages the workspace projection and maps snake_case to the domain shape", async () => { - const axios = { get: vi.fn(async () => ({ data: BACKEND_WORKSPACE })) }; - const page = await new ProjectionRepository(axios as never).getWorkspaceProjection("w-1", 50, 25); + expect(axios.get).toHaveBeenCalledWith("/v2/projection", { + params: { workspace_id: "w-1", offset: 50, limit: 25 }, + }); + expect(page.totalReferences).toBe(213); - expect(axios.get).toHaveBeenCalledWith("/v2/projection", { - params: { workspace_id: "w-1", offset: 50, limit: 25 }, - }); - expect(page.totalReferences).toBe(213); - expect(page.columns[0]).toEqual({ - name: "Design.type", - schemaId: "s-1", - schemaName: "Design", - questionName: "type", - subColumn: null, - dtype: "text", - }); - expect(page.rows[0].reference).toBe("10.1000/j.x"); - expect(page.rows[0].rowIndex).toBe(0); - expect(page.rows[0].cells["Design.type"]).toEqual({ - value: "RCT", - source: "response", - recordId: "r-1", - agent: null, - score: null, + // Ordering invariant: fails if columns are re-ordered/sorted anywhere in the mapping. + expect(page.columns.map((c) => c.name)).toEqual(["Zeta.x", "Alpha.y"]); + expect(page.columns[0]).toEqual({ + name: "Zeta.x", + schemaId: "s-2", + schemaName: "Zeta", + questionName: "x", + subColumn: null, + dtype: "text", + }); + + expect(page.rows[0].reference).toBe("10.1000/j.x"); + expect(page.rows[0].rowIndex).toBe(0); + + // Ordering invariant: fails if cells are rebuilt via a Map/sort that loses server key order. + expect(Object.keys(page.rows[0].cells)).toEqual(["Zeta.x", "Alpha.y"]); + expect(page.rows[0].cells["Zeta.x"]).toEqual({ + value: "RCT", + source: "response", + recordId: "r-1", + agent: null, + score: null, + }); }); - }); - it("defaults to offset 0, limit 50", async () => { - const axios = { get: vi.fn(async () => ({ data: BACKEND_WORKSPACE })) }; - await new ProjectionRepository(axios as never).getWorkspaceProjection("w-1"); - expect(axios.get).toHaveBeenCalledWith("/v2/projection", { - params: { workspace_id: "w-1", offset: 0, limit: 50 }, + it("defaults to offset 0, limit 50", async () => { + const axios = { get: vi.fn(async () => ({ data: BACKEND_WORKSPACE })) }; + await new ProjectionRepository(axios as never).getWorkspaceProjection("w-1"); + expect(axios.get).toHaveBeenCalledWith("/v2/projection", { + params: { workspace_id: "w-1", offset: 0, limit: 50 }, + }); }); }); }); From 86c47d071eb1a5541ca33aa9b367b5e945104537 Mon Sep 17 00:00:00 2001 From: JonnyTran Date: Tue, 21 Jul 2026 12:04:01 -0700 Subject: [PATCH 15/21] test(v2-ui): pin grid-adapter key order and encode annotation URL schema id MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - toPerspectiveData: add a reverse-alphabetical fixture asserting Object.keys() equals [reference, ...manifest order] so a stray alphabetical sort regresses the test, not just the comment. - Cover empty-rows for toPerspectiveData/bandParity, a cells key absent from the manifest, and a reference recurring after a gap ([0,1,0]). - buildAnnotationUrl: encodeURIComponent the schemaId path segment too (not just the reference), plus tests for &/# and a slashed schema id. - Correct the toPerspectiveData comment: the integer-like-key hazard is avoided today because column names always contain a dot, not because of anything the function does — and note Phase 2 should pass projection.columns as Perspective's explicit column config rather than rely on inferred key order. --- .../entities/projection/grid-adapter.test.ts | 68 +++++++++++++++++++ .../entities/projection/grid-adapter.ts | 24 ++++--- 2 files changed, 84 insertions(+), 8 deletions(-) diff --git a/extralit-frontend/v2/domain/entities/projection/grid-adapter.test.ts b/extralit-frontend/v2/domain/entities/projection/grid-adapter.test.ts index 715000628..a275efb57 100644 --- a/extralit-frontend/v2/domain/entities/projection/grid-adapter.test.ts +++ b/extralit-frontend/v2/domain/entities/projection/grid-adapter.test.ts @@ -47,6 +47,48 @@ describe("toPerspectiveData", () => { { [REFERENCE_COLUMN]: "10.1/b", "Design.type": null, "Outcomes.results.value": null }, ]); }); + + it("preserves manifest column order (not alphabetical), reference first", () => { + // Deliberately reverse-alphabetical ("Zebra" before "Apple") so this test fails if + // anyone re-sorts columns instead of following `projection.columns` order. + const orderColumns = [ + { + name: "Zebra.value", + schemaId: "s-2", + schemaName: "Zebra", + questionName: "value", + subColumn: null, + dtype: "text", + }, + { + name: "Apple.value", + schemaId: "s-1", + schemaName: "Apple", + questionName: "value", + subColumn: null, + dtype: "text", + }, + ]; + const orderProjection = new WorkspaceProjection(orderColumns, [{ reference: "10.1/a", rowIndex: 0, cells: {} }], 1); + const rows = toPerspectiveData(orderProjection); + expect(Object.keys(rows[0])).toEqual([REFERENCE_COLUMN, "Zebra.value", "Apple.value"]); + }); + + it("excludes cell keys that are not present in the column manifest", () => { + const extraCellProjection = new WorkspaceProjection( + COLUMNS, + [{ reference: "10.1/a", rowIndex: 0, cells: { "Unlisted.column": cell("x") } }], + 1 + ); + expect(toPerspectiveData(extraCellProjection)).toEqual([ + { [REFERENCE_COLUMN]: "10.1/a", "Design.type": null, "Outcomes.results.value": null }, + ]); + }); + + it("returns an empty array for a projection with no rows", () => { + const empty = new WorkspaceProjection(COLUMNS, [], 0); + expect(toPerspectiveData(empty)).toEqual([]); + }); }); describe("cellAt", () => { @@ -64,6 +106,24 @@ describe("bandParity", () => { it("flips parity when the reference changes, not per row", () => { expect(bandParity(PROJECTION)).toEqual([0, 0, 1]); }); + + it("does not merge a reference that recurs after a gap", () => { + const gapped = new WorkspaceProjection( + COLUMNS, + [ + { reference: "10.1/a", rowIndex: 0, cells: {} }, + { reference: "10.1/b", rowIndex: 0, cells: {} }, + { reference: "10.1/a", rowIndex: 0, cells: {} }, + ], + 2 + ); + expect(bandParity(gapped)).toEqual([0, 1, 0]); + }); + + it("returns an empty array for a projection with no rows", () => { + const empty = new WorkspaceProjection(COLUMNS, [], 0); + expect(bandParity(empty)).toEqual([]); + }); }); describe("annotation URL contract (spec §3.3)", () => { @@ -74,4 +134,12 @@ describe("annotation URL contract (spec §3.3)", () => { it("puts the schema id in the dataset slot and encodes the reference", () => { expect(buildAnnotationUrl("s-1", "10.1/a b")).toBe("/dataset/s-1/annotation-mode?_search=10.1%2Fa%20b"); }); + + it("encodes reserved query characters (&, #) in the reference", () => { + expect(buildAnnotationUrl("s-1", "a&b#c")).toBe("/dataset/s-1/annotation-mode?_search=a%26b%23c"); + }); + + it("encodes reserved characters in the schema id path segment too", () => { + expect(buildAnnotationUrl("s/1", "10.1/a")).toBe("/dataset/s%2F1/annotation-mode?_search=10.1%2Fa"); + }); }); diff --git a/extralit-frontend/v2/domain/entities/projection/grid-adapter.ts b/extralit-frontend/v2/domain/entities/projection/grid-adapter.ts index 7a11aefb5..88ab6c8fc 100644 --- a/extralit-frontend/v2/domain/entities/projection/grid-adapter.ts +++ b/extralit-frontend/v2/domain/entities/projection/grid-adapter.ts @@ -10,11 +10,18 @@ export const REFERENCE_COLUMN = "reference"; * Maps a `WorkspaceProjection` into the flat-row shape Perspective ingests. * * `projection.columns` order is the single source of truth for column order (server-side - * question order is definition order, not alphabetical, and JS object key order is not a - * safe substitute once column names could look numeric). `reference` is emitted first so - * every row shares an identical, stably-ordered key set — every manifest column is present - * on every row (`null` when the cell is absent) so Perspective infers one stable schema - * instead of a per-row-varying one. + * question order is definition order, not alphabetical). We iterate `columns` to build each + * record rather than relying on the emitted plain object's own key insertion order for + * anything downstream: today every column name is `Schema.question[.subcol]` and always + * contains a `.`, so it can never collide with JS's canonical-integer-key hoisting (e.g. a + * bare `"2024"` key would sort before string keys) — but that safety is a property of the + * naming convention, not of this function's return type. A plain `Record` + * has no ordering guarantee Perspective is contractually bound to respect. Handoff note for + * Phase 2: prefer passing `projection.columns` as Perspective's explicit column/schema + * config rather than relying on inferred key order from the objects returned here. + * `reference` is emitted first so every row shares an identical, stably-ordered key set — + * every manifest column is present on every row (`null` when the cell is absent) so + * Perspective infers one stable schema instead of a per-row-varying one. */ export function toPerspectiveData(projection: WorkspaceProjection): Record[] { return projection.rows.map((row) => { @@ -70,9 +77,10 @@ export function bandParity(projection: WorkspaceProjection): number[] { export const ANNOTATION_CELL_LINKS_ENABLED = false; /** - * Builds the annotation-mode deep link for a cell's reference, percent-encoding the - * reference so slashes and other reserved characters survive the query string. + * Builds the annotation-mode deep link for a cell's reference, percent-encoding both the + * schema id (path segment) and the reference (query value) so slashes, `&`, `#`, and other + * reserved characters can't reshape the URL. */ export function buildAnnotationUrl(schemaId: string, reference: string): string { - return `/dataset/${schemaId}/annotation-mode?_search=${encodeURIComponent(reference)}`; + return `/dataset/${encodeURIComponent(schemaId)}/annotation-mode?_search=${encodeURIComponent(reference)}`; } From 2ec8275ee183982a68983e71c766eb3585b7be4f Mon Sep 17 00:00:00 2001 From: JonnyTran Date: Tue, 21 Jul 2026 12:06:23 -0700 Subject: [PATCH 16/21] feat(v2-ui): workspace-projection use-case, extractions storage, DI wiring --- extralit-frontend/v2/di/di.ts | 3 + .../get-workspace-projection-use-case.test.ts | 49 ++++++++++++++++ .../get-workspace-projection-use-case.ts | 56 +++++++++++++++++++ .../storage/ExtractionsStorage.ts | 23 ++++++++ 4 files changed, 131 insertions(+) create mode 100644 extralit-frontend/v2/domain/usecases/get-workspace-projection-use-case.test.ts create mode 100644 extralit-frontend/v2/domain/usecases/get-workspace-projection-use-case.ts create mode 100644 extralit-frontend/v2/infrastructure/storage/ExtractionsStorage.ts diff --git a/extralit-frontend/v2/di/di.ts b/extralit-frontend/v2/di/di.ts index ebb845955..5550877a8 100644 --- a/extralit-frontend/v2/di/di.ts +++ b/extralit-frontend/v2/di/di.ts @@ -8,6 +8,8 @@ import { V2RecordRepository } from "~/v2/infrastructure/repositories/V2RecordRep import { AnnotationRepository } from "~/v2/infrastructure/repositories/AnnotationRepository"; import { ProjectionRepository } from "~/v2/infrastructure/repositories/ProjectionRepository"; import { GetReferenceReviewUseCase } from "~/v2/domain/usecases/get-reference-review-use-case"; +import { GetWorkspaceProjectionUseCase } from "~/v2/domain/usecases/get-workspace-projection-use-case"; +import { useExtractions } from "~/v2/infrastructure/storage/ExtractionsStorage"; import { SubmitReferenceReviewUseCase } from "~/v2/domain/usecases/submit-reference-review-use-case"; import { SaveReviewDraftUseCase } from "~/v2/domain/usecases/save-review-draft-use-case"; import { DiscardReviewUseCase } from "~/v2/domain/usecases/discard-review-use-case"; @@ -42,6 +44,7 @@ export const loadV2DependencyContainer = (nuxtApp: NuxtAppLike) => { register(AnnotationRepository).withDependency(useAxios).build(), register(ProjectionRepository).withDependency(useAxios).build(), + register(GetWorkspaceProjectionUseCase).withDependencies(ProjectionRepository, useExtractions).build(), register(GetReferenceReviewUseCase) .withDependencies( diff --git a/extralit-frontend/v2/domain/usecases/get-workspace-projection-use-case.test.ts b/extralit-frontend/v2/domain/usecases/get-workspace-projection-use-case.test.ts new file mode 100644 index 000000000..cbe95e56d --- /dev/null +++ b/extralit-frontend/v2/domain/usecases/get-workspace-projection-use-case.test.ts @@ -0,0 +1,49 @@ +import { createPinia, setActivePinia } from "pinia"; +import { beforeEach, describe, expect, it, vi } from "vitest"; +import { GetWorkspaceProjectionUseCase, PROJECTION_PAGE_SIZE } from "./get-workspace-projection-use-case"; +import { useExtractions } from "~/v2/infrastructure/storage/ExtractionsStorage"; + +const COLUMN = { + name: "Design.type", + schemaId: "s-1", + schemaName: "Design", + questionName: "type", + subColumn: null, + dtype: "text", +}; + +const row = (reference: string) => ({ reference, rowIndex: 0, cells: {} }); + +describe("GetWorkspaceProjectionUseCase", () => { + beforeEach(() => setActivePinia(createPinia())); + + it("loads every page into one projection and saves it to storage", async () => { + const repository = { + getWorkspaceProjection: vi + .fn() + .mockResolvedValueOnce({ columns: [COLUMN], rows: [row("10.1/a")], totalReferences: 150 }) + .mockResolvedValueOnce({ columns: [COLUMN], rows: [row("10.1/b")], totalReferences: 150 }), + }; + const useCase = new GetWorkspaceProjectionUseCase(repository as never, useExtractions()); + + const projection = await useCase.execute("w-1"); + + expect(repository.getWorkspaceProjection).toHaveBeenNthCalledWith(1, "w-1", 0, PROJECTION_PAGE_SIZE); + expect(repository.getWorkspaceProjection).toHaveBeenNthCalledWith(2, "w-1", 100, PROJECTION_PAGE_SIZE); + expect(repository.getWorkspaceProjection).toHaveBeenCalledTimes(2); + expect(projection.rows.map((r) => r.reference)).toEqual(["10.1/a", "10.1/b"]); + expect(projection.totalReferences).toBe(150); + expect(useExtractions().get().projection).toEqual(projection); + }); + + it("makes a single call when everything fits in one page", async () => { + const repository = { + getWorkspaceProjection: vi + .fn() + .mockResolvedValue({ columns: [COLUMN], rows: [row("10.1/a")], totalReferences: 1 }), + }; + const useCase = new GetWorkspaceProjectionUseCase(repository as never, useExtractions()); + await useCase.execute("w-1"); + expect(repository.getWorkspaceProjection).toHaveBeenCalledTimes(1); + }); +}); diff --git a/extralit-frontend/v2/domain/usecases/get-workspace-projection-use-case.ts b/extralit-frontend/v2/domain/usecases/get-workspace-projection-use-case.ts new file mode 100644 index 000000000..3e10d4670 --- /dev/null +++ b/extralit-frontend/v2/domain/usecases/get-workspace-projection-use-case.ts @@ -0,0 +1,56 @@ +import { ProjectionRepository } from "~/v2/infrastructure/repositories/ProjectionRepository"; +import { + WorkspaceProjection, + type ProjectionColumn, + type ProjectionGridRow, +} from "~/v2/domain/entities/projection/WorkspaceProjection"; +import { type useExtractions } from "~/v2/infrastructure/storage/ExtractionsStorage"; + +// The server's `limit` cap per page. +export const PROJECTION_PAGE_SIZE = 100; + +export class GetWorkspaceProjectionUseCase { + constructor( + private readonly projectionRepository: ProjectionRepository, + // ts-injecty resolves the `useExtractions` hook by calling it, so the injected + // value is the store object, not the hook (same contract as GetReferenceReviewUseCase). + private readonly extractionsStorage: ReturnType + ) {} + + // Loads the whole workspace projection in one shot: the grid renders as a single + // Perspective table client-side, so it needs every row up front rather than a page + // at a time. Streaming this via Arrow IPC is the recorded §5 follow-up. + async execute(workspaceId: string): Promise { + let offset = 0; + let columns: ProjectionColumn[] = []; + let totalReferences = 0; + const rows: ProjectionGridRow[] = []; + + for (;;) { + const page = await this.projectionRepository.getWorkspaceProjection(workspaceId, offset, PROJECTION_PAGE_SIZE); + if (offset === 0) { + // Every page returns the same manifest — capture it once, in server order. + columns = page.columns; + } + totalReferences = page.totalReferences; + rows.push(...page.rows); + + // Guard against a zero-progress response (e.g. a server bug that keeps reporting + // outstanding references but never returns any) so a stuck server can't hang the browser. + if (page.rows.length === 0) { + break; + } + + // `offset`/`limit` count REFERENCES, not the fanned-out rows the endpoint returns — + // always advance by the fixed page size, never by `rows.length`. + offset += PROJECTION_PAGE_SIZE; + if (offset >= totalReferences) { + break; + } + } + + const projection = new WorkspaceProjection(columns, rows, totalReferences); + this.extractionsStorage.saveProjection(projection); + return projection; + } +} diff --git a/extralit-frontend/v2/infrastructure/storage/ExtractionsStorage.ts b/extralit-frontend/v2/infrastructure/storage/ExtractionsStorage.ts new file mode 100644 index 000000000..1a7ab14bf --- /dev/null +++ b/extralit-frontend/v2/infrastructure/storage/ExtractionsStorage.ts @@ -0,0 +1,23 @@ +import { useStoreFor } from "@/v1/store/create"; +import { WorkspaceProjection } from "~/v2/domain/entities/projection/WorkspaceProjection"; + +// Class name is the Pinia store key — must stay unique vs every v1/v2 useStoreFor class. +class Extractions { + constructor(public readonly projection: WorkspaceProjection | null = null) {} +} + +interface IExtractionsStorage { + saveProjection(projection: WorkspaceProjection): void; +} + +const useStoreForExtractions = useStoreFor(Extractions); + +export const useExtractions = () => { + const store = useStoreForExtractions(); + + const saveProjection = (projection: WorkspaceProjection) => { + store.save(new Extractions(projection)); + }; + + return { ...store, saveProjection }; +}; From 989190c09c66a35d2f829435934149d5d058bf9c Mon Sep 17 00:00:00 2001 From: JonnyTran Date: Tue, 21 Jul 2026 12:15:33 -0700 Subject: [PATCH 17/21] test(v2-ui): pin manifest aggregation and zero-progress guard Pins the two hazards the paging fix relies on but that were previously unasserted: (1) columns are captured once from the first page, in server order, never sorted or duplicated across pages; (2) an empty page with a huge totalReferences stops after one call instead of hammering the endpoint. Also captures totalReferences from the first page only (matching columns), and documents the server spine-CTE invariant the zero-progress guard depends on. --- .../get-workspace-projection-use-case.test.ts | 45 +++++++++++++++++++ .../get-workspace-projection-use-case.ts | 9 +++- 2 files changed, 52 insertions(+), 2 deletions(-) diff --git a/extralit-frontend/v2/domain/usecases/get-workspace-projection-use-case.test.ts b/extralit-frontend/v2/domain/usecases/get-workspace-projection-use-case.test.ts index cbe95e56d..ec23cb9d2 100644 --- a/extralit-frontend/v2/domain/usecases/get-workspace-projection-use-case.test.ts +++ b/extralit-frontend/v2/domain/usecases/get-workspace-projection-use-case.test.ts @@ -12,6 +12,25 @@ const COLUMN = { dtype: "text", }; +// Deliberately reverse-alphabetical so a `.sort()` regression on the manifest would be caught. +const COLUMN_ZEBRA = { + name: "Zebra.count", + schemaId: "s-2", + schemaName: "Zebra", + questionName: "count", + subColumn: null, + dtype: "number", +}; +const COLUMN_APPLE = { + name: "Apple.type", + schemaId: "s-3", + schemaName: "Apple", + questionName: "type", + subColumn: null, + dtype: "text", +}; +const SERVER_ORDER_COLUMNS = [COLUMN_ZEBRA, COLUMN_APPLE]; + const row = (reference: string) => ({ reference, rowIndex: 0, cells: {} }); describe("GetWorkspaceProjectionUseCase", () => { @@ -46,4 +65,30 @@ describe("GetWorkspaceProjectionUseCase", () => { await useCase.execute("w-1"); expect(repository.getWorkspaceProjection).toHaveBeenCalledTimes(1); }); + + it("aggregates the manifest once, in server order, never sorted or duplicated", async () => { + const repository = { + getWorkspaceProjection: vi + .fn() + .mockResolvedValueOnce({ columns: SERVER_ORDER_COLUMNS, rows: [row("10.1/a")], totalReferences: 150 }) + .mockResolvedValueOnce({ columns: SERVER_ORDER_COLUMNS, rows: [row("10.1/b")], totalReferences: 150 }), + }; + const useCase = new GetWorkspaceProjectionUseCase(repository as never, useExtractions()); + + const projection = await useCase.execute("w-1"); + + expect(projection.columns).toEqual(SERVER_ORDER_COLUMNS); + }); + + it("stops after a single call on a zero-progress response instead of hanging", async () => { + const repository = { + getWorkspaceProjection: vi.fn().mockResolvedValue({ columns: [COLUMN], rows: [], totalReferences: 999999 }), + }; + const useCase = new GetWorkspaceProjectionUseCase(repository as never, useExtractions()); + + const projection = await useCase.execute("w-1"); + + expect(repository.getWorkspaceProjection).toHaveBeenCalledTimes(1); + expect(projection.rows).toEqual([]); + }); }); diff --git a/extralit-frontend/v2/domain/usecases/get-workspace-projection-use-case.ts b/extralit-frontend/v2/domain/usecases/get-workspace-projection-use-case.ts index 3e10d4670..e68302fd3 100644 --- a/extralit-frontend/v2/domain/usecases/get-workspace-projection-use-case.ts +++ b/extralit-frontend/v2/domain/usecases/get-workspace-projection-use-case.ts @@ -29,14 +29,19 @@ export class GetWorkspaceProjectionUseCase { for (;;) { const page = await this.projectionRepository.getWorkspaceProjection(workspaceId, offset, PROJECTION_PAGE_SIZE); if (offset === 0) { - // Every page returns the same manifest — capture it once, in server order. + // Every page returns the same manifest and total — capture both from the first + // page only, in server order, so a shifting count mid-paging can't mix into a + // result built from a stale row set. columns = page.columns; + totalReferences = page.totalReferences; } - totalReferences = page.totalReferences; rows.push(...page.rows); // Guard against a zero-progress response (e.g. a server bug that keeps reporting // outstanding references but never returns any) so a stuck server can't hang the browser. + // Safe because every reference in a page yields >= 1 row: the server's `spine` CTE + // emits `max_idx + 1` rows per reference (minimum 1) and keeps the spine row even when + // `column_name IS NULL` — so an empty `rows` array can only mean an empty reference page. if (page.rows.length === 0) { break; } From 2a1ac44f7a6339233cff9590b866c7b86c2e3c77 Mon Sep 17 00:00:00 2001 From: JonnyTran Date: Tue, 21 Jul 2026 14:07:50 -0700 Subject: [PATCH 18/21] fix(v2): exempt table values from suggestion score cardinality; close review gaps Addresses medium-severity roborev findings on the Phase 1 branch. Behavior fix: - V2SuggestionValidator._validate_score now takes the question type and skips the list-cardinality rules for table questions. Those rules assume a list value is a list of answer choices (multi_label_selection, ranking); a table value's list is N rows (spec 3.4), and a suggestion's score applies to the suggestion as a whole. Before this, a multi-row table suggestion with one confidence score -- the exact shape Task 3's fan-out consumes -- was rejected by upsert_suggestion. Hardening: - Serialize DuckDB inputs with ensure_ascii=False. The projection joins Python strings against keys DuckDB parses from JSON text; emitting non-ASCII names as \uXXXX made that join depend on DuckDB decoding them back. It does decode them correctly today, so this removes a dependency rather than fixing a live break. Test gaps closed: - table suggestions: single score over multi-row value, mismatched score list, and score list over a bare dict; plus non-table types still enforcing cardinality - multi-question response envelope: pins the positional key/value zip-unnest that every other test exercised with only one key - non-ASCII schema/question/sub-column names resolve end to end - response-over-suggestion cell: seed agent/score on the losing suggestion so the `is None` assertions prove suppression instead of passing vacuously - repository fixture: suggestion-sourced table sub-column with agent/score and a list-valued score, so a transposition or dropped sub_column now fails - use-case: first-page totalReferences capture (growing total mid-sweep) and the empty-workspace path; manifest test pages now return distinct manifests Server 138 passed; frontend 871 passed, typecheck/lint/gen:api drift all clean. --- .../get-workspace-projection-use-case.test.ts | 40 ++++++++++- .../repositories/ProjectionRepository.test.ts | 67 ++++++++++++++++--- .../extralit_server/contexts/v2/projection.py | 17 ++++- .../extralit_server/validators/v2/values.py | 11 ++- .../contexts/v2/test_projection.py | 4 +- .../contexts/v2/test_workspace_projection.py | 55 +++++++++++++++ .../tests/unit/validators/v2/test_values.py | 48 +++++++++++++ 7 files changed, 226 insertions(+), 16 deletions(-) diff --git a/extralit-frontend/v2/domain/usecases/get-workspace-projection-use-case.test.ts b/extralit-frontend/v2/domain/usecases/get-workspace-projection-use-case.test.ts index ec23cb9d2..8c3a6b962 100644 --- a/extralit-frontend/v2/domain/usecases/get-workspace-projection-use-case.test.ts +++ b/extralit-frontend/v2/domain/usecases/get-workspace-projection-use-case.test.ts @@ -70,8 +70,11 @@ describe("GetWorkspaceProjectionUseCase", () => { const repository = { getWorkspaceProjection: vi .fn() + // Page 2 returns a DIFFERENT manifest: if the manifest were captured from the last + // page (or reassigned every page) rather than the first, this test fails. Sharing one + // array across both pages could not distinguish those implementations. .mockResolvedValueOnce({ columns: SERVER_ORDER_COLUMNS, rows: [row("10.1/a")], totalReferences: 150 }) - .mockResolvedValueOnce({ columns: SERVER_ORDER_COLUMNS, rows: [row("10.1/b")], totalReferences: 150 }), + .mockResolvedValueOnce({ columns: [COLUMN], rows: [row("10.1/b")], totalReferences: 150 }), }; const useCase = new GetWorkspaceProjectionUseCase(repository as never, useExtractions()); @@ -80,6 +83,41 @@ describe("GetWorkspaceProjectionUseCase", () => { expect(projection.columns).toEqual(SERVER_ORDER_COLUMNS); }); + it("fixes the page count at sweep start so a growing total cannot run away", async () => { + // Records landing mid-sweep raise the server's reported total. The loop bound must be + // whatever the first page reported, or the sweep chases a moving target. The zero-progress + // guard cannot catch this: every page here returns a non-empty rows array. + const repository = { + getWorkspaceProjection: vi + .fn() + .mockResolvedValueOnce({ columns: [COLUMN], rows: [row("10.1/a")], totalReferences: 150 }) + .mockResolvedValueOnce({ columns: [COLUMN], rows: [row("10.1/b")], totalReferences: 350 }) + .mockResolvedValue({ columns: [COLUMN], rows: [row("10.1/c")], totalReferences: 350 }), + }; + const useCase = new GetWorkspaceProjectionUseCase(repository as never, useExtractions()); + + const projection = await useCase.execute("w-1"); + + expect(repository.getWorkspaceProjection).toHaveBeenCalledTimes(2); + expect(projection.totalReferences).toBe(150); + expect(projection.rows.map((r) => r.reference)).toEqual(["10.1/a", "10.1/b"]); + }); + + it("returns an empty projection with the manifest for a workspace with no records", async () => { + // Schemas but no records — the empty state `/extractions` must render in Phase 2. + const repository = { + getWorkspaceProjection: vi.fn().mockResolvedValue({ columns: [COLUMN], rows: [], totalReferences: 0 }), + }; + const useCase = new GetWorkspaceProjectionUseCase(repository as never, useExtractions()); + + const projection = await useCase.execute("w-1"); + + expect(repository.getWorkspaceProjection).toHaveBeenCalledTimes(1); + expect(projection.rows).toEqual([]); + expect(projection.columns).toEqual([COLUMN]); + expect(projection.totalReferences).toBe(0); + }); + it("stops after a single call on a zero-progress response instead of hanging", async () => { const repository = { getWorkspaceProjection: vi.fn().mockResolvedValue({ columns: [COLUMN], rows: [], totalReferences: 999999 }), diff --git a/extralit-frontend/v2/infrastructure/repositories/ProjectionRepository.test.ts b/extralit-frontend/v2/infrastructure/repositories/ProjectionRepository.test.ts index 9ad9dcbd7..38f462140 100644 --- a/extralit-frontend/v2/infrastructure/repositories/ProjectionRepository.test.ts +++ b/extralit-frontend/v2/infrastructure/repositories/ProjectionRepository.test.ts @@ -18,9 +18,12 @@ const BACKEND_VIEW = { ], }; -// Columns deliberately out of alphabetical order — the server orders columns by schema/question -// definition order, not alphabetically, and nothing on the server pins that order. The grid's -// column layout depends on it, so the mapping must preserve exactly what the server sent. +// The server pins column order via `Schema.name`, then `V2Question.inserted_at, V2Question.name` +// (contexts/v2/projection.py) — so ordering IS deterministic server-side, but it is definition +// order within a schema, not alphabetical overall. This fixture is deliberately arranged +// non-alphabetically so an accidental client-side sort fails the order assertions below. +// It also covers the enriched-provenance fields end-to-end: a suggestion-sourced table +// sub-column with agent/score, and a list-valued score, which the contract permits. const BACKEND_WORKSPACE = { columns: [ { @@ -32,12 +35,20 @@ const BACKEND_WORKSPACE = { dtype: "text", }, { - name: "Alpha.y", + name: "Alpha.results.value", schema_id: "s-1", schema_name: "Alpha", - question_name: "y", + question_name: "results", + sub_column: "value", + dtype: "table", + }, + { + name: "Alpha.labels", + schema_id: "s-1", + schema_name: "Alpha", + question_name: "labels", sub_column: null, - dtype: "text", + dtype: "multi_label_selection", }, ], rows: [ @@ -46,7 +57,20 @@ const BACKEND_WORKSPACE = { row_index: 0, cells: { "Zeta.x": { value: "RCT", source: "response", record_id: "r-1", agent: null, score: null }, - "Alpha.y": { value: "42", source: "response", record_id: "r-2", agent: null, score: null }, + "Alpha.results.value": { + value: "12%", + source: "suggestion", + record_id: "r-2", + agent: "gpt-x", + score: 0.92, + }, + "Alpha.labels": { + value: ["a", "b"], + source: "suggestion", + record_id: "r-2", + agent: "gpt-x", + score: [0.9, 0.1], + }, }, }, ], @@ -90,7 +114,7 @@ describe("ProjectionRepository", () => { expect(page.totalReferences).toBe(213); // Ordering invariant: fails if columns are re-ordered/sorted anywhere in the mapping. - expect(page.columns.map((c) => c.name)).toEqual(["Zeta.x", "Alpha.y"]); + expect(page.columns.map((c) => c.name)).toEqual(["Zeta.x", "Alpha.results.value", "Alpha.labels"]); expect(page.columns[0]).toEqual({ name: "Zeta.x", schemaId: "s-2", @@ -103,8 +127,10 @@ describe("ProjectionRepository", () => { expect(page.rows[0].reference).toBe("10.1000/j.x"); expect(page.rows[0].rowIndex).toBe(0); - // Ordering invariant: fails if cells are rebuilt via a Map/sort that loses server key order. - expect(Object.keys(page.rows[0].cells)).toEqual(["Zeta.x", "Alpha.y"]); + // Cell key order is incidental, not contractual — `toPerspectiveData` builds each row by + // iterating `projection.columns`, never `row.cells` key order. Asserted only to document + // that the mapping passes the server's object through rather than rebuilding it. + expect(Object.keys(page.rows[0].cells)).toEqual(["Zeta.x", "Alpha.results.value", "Alpha.labels"]); expect(page.rows[0].cells["Zeta.x"]).toEqual({ value: "RCT", source: "response", @@ -112,6 +138,27 @@ describe("ProjectionRepository", () => { agent: null, score: null, }); + + // Enriched provenance: a transposition (agent<->score) or a dropped sub_column would + // pass if every fixture cell null-filled these, so assert them populated. + expect(page.columns[1].subColumn).toBe("value"); + expect(page.columns[1].dtype).toBe("table"); + expect(page.columns[2].subColumn).toBeNull(); + expect(page.rows[0].cells["Alpha.results.value"]).toEqual({ + value: "12%", + source: "suggestion", + recordId: "r-2", + agent: "gpt-x", + score: 0.92, + }); + // The contract permits a list-valued score; it must survive unchanged. + expect(page.rows[0].cells["Alpha.labels"]).toEqual({ + value: ["a", "b"], + source: "suggestion", + recordId: "r-2", + agent: "gpt-x", + score: [0.9, 0.1], + }); }); it("defaults to offset 0, limit 50", async () => { diff --git a/extralit-server/src/extralit_server/contexts/v2/projection.py b/extralit-server/src/extralit_server/contexts/v2/projection.py index 96ec9610a..0f062acc0 100644 --- a/extralit-server/src/extralit_server/contexts/v2/projection.py +++ b/extralit-server/src/extralit_server/contexts/v2/projection.py @@ -388,11 +388,24 @@ async def build_workspace_view(db: AsyncSession, *, workspace_id: UUID, offset: (str(q.id), sub) for q in questions if q.type == QuestionType.table for sub in (q.columns or []) ], "records": [(str(r.id), str(r.schema_id), r.reference, r.inserted_at) for r in records], + # ensure_ascii=False: question names and sub-column bindings are matched by string + # equality against keys DuckDB parses out of this JSON text (`rc.question_name = + # q.question_name`, `qc.sub_column = e.entry_key`). Emitting non-ASCII keys as + # \uXXXX escapes would make that join depend on DuckDB decoding them back; writing + # the codepoints directly removes the dependency instead of relying on it. "suggestions": [ - (str(s.record_id), str(s.question_id), json.dumps(s.value), s.agent, json.dumps(s.score)) + ( + str(s.record_id), + str(s.question_id), + json.dumps(s.value, ensure_ascii=False), + s.agent, + json.dumps(s.score, ensure_ascii=False), + ) for s in suggestions ], - "responses": [(str(r.record_id), json.dumps(r.values or {}), r.updated_at) for r in responses], + "responses": [ + (str(r.record_id), json.dumps(r.values or {}, ensure_ascii=False), r.updated_at) for r in responses + ], } output = await to_thread.run_sync(_run_denormalization, inputs) diff --git a/extralit-server/src/extralit_server/validators/v2/values.py b/extralit-server/src/extralit_server/validators/v2/values.py index 3bc48cc19..6c74d94b9 100644 --- a/extralit-server/src/extralit_server/validators/v2/values.py +++ b/extralit-server/src/extralit_server/validators/v2/values.py @@ -66,10 +66,17 @@ class V2SuggestionValidator: @classmethod def validate(cls, value, score, *, type: QuestionType, settings: dict, columns: list[str]) -> None: V2ResponseValueValidator.validate(value, type=type, settings=settings, columns=columns) - cls._validate_score(value, score) + cls._validate_score(value, score, type=type) @staticmethod - def _validate_score(value, score) -> None: + def _validate_score(value, score, *, type: QuestionType) -> None: + if type == QuestionType.table: + # The cardinality rules below assume a list value is a list of *answer choices* + # (multi_label_selection, ranking), where one score per choice is meaningful. + # A table value's list is N *rows* (spec §3.4), and a suggestion's score applies + # to the suggestion as a whole — the projection fan-out repeats it onto every + # fanned-out cell. Row count and score count are unrelated, so skip the checks. + return if not isinstance(value, list) and isinstance(score, list): raise UnprocessableEntityError("a list of scores is not allowed for a single-value suggestion") if isinstance(value, list) and score is not None and not isinstance(score, list): diff --git a/extralit-server/tests/integration/contexts/v2/test_projection.py b/extralit-server/tests/integration/contexts/v2/test_projection.py index 5687b2dad..ec7a08155 100644 --- a/extralit-server/tests/integration/contexts/v2/test_projection.py +++ b/extralit-server/tests/integration/contexts/v2/test_projection.py @@ -45,7 +45,9 @@ async def test_cell_resolves_to_suggestion_when_no_response(db): async def test_cell_resolves_to_response_over_suggestion(db): schema, version, q = await _schema_with_question(db) record = await V2RecordFactory.create(version=version, reference="doc-2") - await V2SuggestionFactory.create(record=record, question=q, value="flu") + # Seed provenance on the losing suggestion so the `is None` assertions below actually + # prove the response branch suppresses it, rather than passing vacuously. + await V2SuggestionFactory.create(record=record, question=q, value="flu", agent="gpt-x", score=0.92) user = await UserFactory.create() await V2ResponseFactory.create( record=record, user=user, status=ResponseStatus.submitted, values={"dx": {"value": "covid"}} diff --git a/extralit-server/tests/integration/contexts/v2/test_workspace_projection.py b/extralit-server/tests/integration/contexts/v2/test_workspace_projection.py index fff5af555..7d403b088 100644 --- a/extralit-server/tests/integration/contexts/v2/test_workspace_projection.py +++ b/extralit-server/tests/integration/contexts/v2/test_workspace_projection.py @@ -240,3 +240,58 @@ async def counting_execute(*args, **kwargs): await projection_ctx.build_workspace_view(db, workspace_id=workspace.id, offset=0, limit=50) # schemas, questions, ref-count, ref-page, records, suggestions, responses => 7 max assert len(executed) <= 7, f"N+1 regression: {len(executed)} statements" + + +async def test_multi_question_response_envelope_attributes_each_value_to_its_own_question(db): + # The response path pairs json_keys(values_json) with json_extract(values_json, '$.*') + # positionally and then joins on question_name. Every other test in this file submits a + # single-key envelope, so a misalignment would be invisible. This is the real-world + # shape: one user answering several questions on one record. + workspace = await WorkspaceFactory.create() + schema, version = await _make_schema(workspace, "Design") + await _add_question(schema, "type") + await _add_question(schema, "country") + await _add_question(schema, "notes") + rec = await V2RecordFactory.create(version=version, reference="10.1/a") + user = await UserFactory.create() + await V2ResponseFactory.create( + record=rec, + user=user, + status=ResponseStatus.submitted, + values={ + "type": {"value": "RCT"}, + "country": {"value": "KE"}, + "notes": {"value": "multi-site"}, + }, + ) + + view = await projection_ctx.build_workspace_view(db, workspace_id=workspace.id, offset=0, limit=50) + + cells = view.rows[0].cells + assert cells["Design.type"].value == "RCT" + assert cells["Design.country"].value == "KE" + assert cells["Design.notes"].value == "multi-site" + assert all(cells[name].source == "response" for name in ("Design.type", "Design.country", "Design.notes")) + + +async def test_non_ascii_names_and_bindings_resolve(db): + # Both joins compare Python strings against keys DuckDB parsed out of JSON text. + # Non-ASCII names must survive that round-trip (see ensure_ascii=False at the input + # serialization) — a mismatch would silently omit the cell rather than error. + workspace = await WorkspaceFactory.create() + schema, version = await _make_schema(workspace, "Résumé") + await _add_question(schema, "pays") + table_q = await _add_question(schema, "résultats", qtype=QuestionType.table, columns=["café", "日本語"]) + rec = await V2RecordFactory.create(version=version, reference="10.1/a") + user = await UserFactory.create() + await V2SuggestionFactory.create(record=rec, question=table_q, value=[{"café": "noir", "日本語": "はい"}]) + await V2ResponseFactory.create( + record=rec, user=user, status=ResponseStatus.submitted, values={"pays": {"value": "Côte d'Ivoire"}} + ) + + view = await projection_ctx.build_workspace_view(db, workspace_id=workspace.id, offset=0, limit=50) + + cells = view.rows[0].cells + assert cells["Résumé.pays"].value == "Côte d'Ivoire" + assert cells["Résumé.résultats.café"].value == "noir" + assert cells["Résumé.résultats.日本語"].value == "はい" diff --git a/extralit-server/tests/unit/validators/v2/test_values.py b/extralit-server/tests/unit/validators/v2/test_values.py index 40b02f035..2c85c493e 100644 --- a/extralit-server/tests/unit/validators/v2/test_values.py +++ b/extralit-server/tests/unit/validators/v2/test_values.py @@ -84,3 +84,51 @@ def test_table_value_list_rejects_non_dict_rows(): V2ResponseValueValidator.validate( [{"a": 1}, 5], type=QuestionType.table, settings=TABLE_SETTINGS, columns=["a"] ) + + +def test_table_suggestion_allows_single_score_for_multi_row_value(): + # A table value's list is N rows (spec §3.4), not N answer choices: one confidence + # score describes the whole suggestion and the projection fan-out repeats it onto + # every fanned-out cell. The list-cardinality rules must not apply here. + V2SuggestionValidator.validate( + [{"value": "12%"}, {"value": "8%"}], + 0.92, + type=QuestionType.table, + settings=TABLE_SETTINGS, + columns=["value"], + ) + + +def test_table_suggestion_allows_score_list_not_matching_row_count(): + V2SuggestionValidator.validate( + [{"value": "12%"}, {"value": "8%"}, {"value": "3%"}], + [0.9, 0.1], + type=QuestionType.table, + settings=TABLE_SETTINGS, + columns=["value"], + ) + + +def test_table_suggestion_allows_score_list_for_bare_dict_value(): + V2SuggestionValidator.validate( + {"value": "12%"}, [0.9, 0.1], type=QuestionType.table, settings=TABLE_SETTINGS, columns=["value"] + ) + + +def test_non_table_suggestion_still_rejects_single_score_for_list_value(): + # The exemption above must be scoped to table only — these rules still bind elsewhere. + with pytest.raises(UnprocessableEntityError, match="a single score is not allowed"): + V2SuggestionValidator.validate( + ["yes", "no"], 0.9, type=QuestionType.multi_label_selection, settings=LABEL_SETTINGS, columns=["q"] + ) + + +def test_non_table_suggestion_still_rejects_mismatched_score_count(): + with pytest.raises(UnprocessableEntityError, match="doesn't match"): + V2SuggestionValidator.validate( + ["yes", "no"], + [0.9], + type=QuestionType.multi_label_selection, + settings=LABEL_SETTINGS, + columns=["q"], + ) From 06f7e6808ba1b9bdb95d9dd2e9f90188a2ac227c Mon Sep 17 00:00:00 2001 From: JonnyTran Date: Tue, 21 Jul 2026 16:38:27 -0700 Subject: [PATCH 19/21] chore: update VSCode settings for cursorpyright analysis paths and clean up SQL denormalization logic - Added extraPaths for cursorpyright analysis to match Python analysis settings. - Simplified the SQL denormalization logic by replacing the detailed SQL statement with a placeholder, indicating that the exact SQL is to be defined by the implementer. --- .vscode/settings.json | 4 + .../plans/2026-07-20-extraction-table.md | 767 ++---------------- 2 files changed, 64 insertions(+), 707 deletions(-) diff --git a/.vscode/settings.json b/.vscode/settings.json index 8b3cd1e1a..57397a3b6 100644 --- a/.vscode/settings.json +++ b/.vscode/settings.json @@ -33,5 +33,9 @@ "python.analysis.extraPaths": [ "${workspaceFolder}/extralit-server/src", "${workspaceFolder}/extralit/src" + ], + "cursorpyright.analysis.extraPaths": [ + "${workspaceFolder}/extralit-server/src", + "${workspaceFolder}/extralit/src" ] } \ No newline at end of file diff --git a/docs/superpowers/plans/2026-07-20-extraction-table.md b/docs/superpowers/plans/2026-07-20-extraction-table.md index 2c67cb81d..829982130 100644 --- a/docs/superpowers/plans/2026-07-20-extraction-table.md +++ b/docs/superpowers/plans/2026-07-20-extraction-table.md @@ -577,248 +577,49 @@ def _build_columns( return columns -# One SQL statement performs the entire denormalization (spec §3.1/§3.4): effective- -# record dedup, response ?? suggestion coalesce, table fan-out, the independent-stacking -# row spine, and scalar repetition. Long format out: one row per (reference, row_idx, -# column) — plus a bare spine row when a reference has no resolved cells yet. -_DENORMALIZE_SQL = """ -WITH effective_records AS ( -- one effective record per (reference, schema): latest inserted_at - SELECT record_id, schema_id, reference - FROM records - QUALIFY row_number() OVER (PARTITION BY reference, schema_id ORDER BY inserted_at DESC) = 1 -), -latest_responses AS ( -- latest submitted response per record, ANY user (user-agnostic view) - SELECT record_id, values_json - FROM responses - QUALIFY row_number() OVER (PARTITION BY record_id ORDER BY updated_at DESC) = 1 -), -response_cells AS ( -- explode the {question_name: {"value": ...}} envelope - SELECT lr.record_id, je.key AS question_name, json_extract(je.value, '$.value') AS value - FROM latest_responses lr, json_each(lr.values_json) je -), -resolved AS ( -- coalesce: response ?? suggestion; (record, question) with neither drops out - SELECT er.reference, er.record_id, q.question_id, q.schema_name, q.question_name, q.qtype, - COALESCE(rc.value, s.value_json) AS value, - CASE WHEN rc.value IS NOT NULL THEN 'response' ELSE 'suggestion' END AS source, - CASE WHEN rc.value IS NULL THEN s.agent END AS agent, - CASE WHEN rc.value IS NULL THEN s.score_json END AS score - FROM effective_records er - JOIN questions q ON q.schema_id = er.schema_id - LEFT JOIN response_cells rc - ON rc.record_id = er.record_id AND rc.question_name = q.question_name - LEFT JOIN suggestions s - ON s.record_id = er.record_id AND s.question_id = q.question_id - WHERE COALESCE(rc.value, s.value_json) IS NOT NULL -), -table_rows AS ( -- §3.4 in SQL: bare dict = the 1-row case; arrays fan out with 0-based indices - SELECT r.reference, r.record_id, r.question_id, r.schema_name, r.question_name, - r.source, r.agent, r.score, - CAST(je.key AS BIGINT) AS row_idx, je.value AS row_json - FROM resolved r, - json_each(CASE WHEN json_type(r.value) = 'ARRAY' THEN r.value - ELSE json_array(r.value) END) je - WHERE r.qtype = 'table' AND json_type(je.value) = 'OBJECT' -), -table_cells AS ( -- fan each row dict out over the question's bound sub-columns - SELECT tr.reference, tr.row_idx, - tr.schema_name || '.' || tr.question_name || '.' || qc.sub_column AS column_name, - json_extract(tr.row_json, '$."' || qc.sub_column || '"') AS value, - tr.source, tr.record_id, tr.agent, tr.score - FROM table_rows tr - JOIN question_columns qc ON qc.question_id = tr.question_id - WHERE COALESCE(json_type(json_extract(tr.row_json, '$."' || qc.sub_column || '"')), 'NULL') <> 'NULL' -), -scalar_cells AS ( - SELECT reference, schema_name || '.' || question_name AS column_name, - value, source, record_id, agent, score - FROM resolved - WHERE qtype <> 'table' AND json_type(value) <> 'NULL' -), -spine AS ( -- independent stacking: rows per reference = max table fan-out, min 1 - SELECT refs.reference, unnest(generate_series(0, COALESCE(mx.mx, 0))) AS row_idx - FROM (SELECT DISTINCT reference FROM effective_records) refs - LEFT JOIN (SELECT reference, MAX(row_idx) AS mx FROM table_rows GROUP BY reference) mx - ON mx.reference = refs.reference -), -long_cells AS ( - SELECT sp.reference, sp.row_idx, sc.column_name, sc.value, - sc.source, sc.record_id, sc.agent, sc.score - FROM spine sp - JOIN scalar_cells sc ON sc.reference = sp.reference -- scalar repetition on every fan-out row - UNION ALL - SELECT reference, row_idx, column_name, value, source, record_id, agent, score - FROM table_cells -) -SELECT sp.reference, sp.row_idx, lc.column_name, CAST(lc.value AS VARCHAR) AS value_json, - lc.source, lc.record_id, lc.agent, CAST(lc.score AS VARCHAR) AS score_json -FROM spine sp -LEFT JOIN long_cells lc ON lc.reference = sp.reference AND lc.row_idx = sp.row_idx -ORDER BY sp.reference, sp.row_idx, lc.column_name -""" - - -_INPUT_DDL = { - "questions": "question_id VARCHAR, schema_id VARCHAR, schema_name VARCHAR, question_name VARCHAR, qtype VARCHAR", - "question_columns": "question_id VARCHAR, sub_column VARCHAR", - "records": "record_id VARCHAR, schema_id VARCHAR, reference VARCHAR, inserted_at TIMESTAMP", - "suggestions": "record_id VARCHAR, question_id VARCHAR, value_json JSON, agent VARCHAR, score_json JSON", - "responses": "record_id VARCHAR, values_json JSON, updated_at TIMESTAMP", -} +_DENORMALIZE_SQL = """...""" # ONE statement; CTE outline below — exact SQL is the implementer's call def _run_denormalization(inputs: dict[str, list[tuple]]) -> list[tuple]: - """Feed the raw slices into an in-memory DuckDB and run the denormalization. - Synchronous/CPU-bound — callers offload via to_thread. NEVER attach Postgres from - here: integration tests run inside a rolled-back transaction a second connection - cannot see, and it would add a second credential path.""" - con = duckdb.connect() - try: - for table, ddl in _INPUT_DDL.items(): - con.execute(f"CREATE TABLE {table}({ddl})") - rows = inputs[table] - if rows: - placeholders = ", ".join("?" * len(rows[0])) - con.executemany(f"INSERT INTO {table} VALUES ({placeholders})", rows) - return con.execute(_DENORMALIZE_SQL).fetchall() - finally: - con.close() + """CREATE + executemany the five input tables into an in-memory duckdb.connect(), + run _DENORMALIZE_SQL, fetchall(), close.""" async def build_workspace_view( db: AsyncSession, *, workspace_id: UUID, offset: int, limit: int ) -> WorkspaceProjection: - """Workspace-level denormalized projection (spec §3.2). Postgres serves batched raw - slices — a fixed number of queries regardless of reference/record count — and the - DuckDB statement above does all the semantics. `offset`/`limit` count references.""" - schemas = ( - (await db.execute(select(Schema).where(Schema.workspace_id == workspace_id).order_by(Schema.name))) - .scalars() - .all() - ) - schema_ids = [s.id for s in schemas] - - questions_by_schema: dict[UUID, list[V2Question]] = {} - if schema_ids: - question_rows = ( - ( - await db.execute( - select(V2Question) - .where(V2Question.schema_id.in_(schema_ids)) - .order_by(V2Question.inserted_at) - ) - ) - .scalars() - .all() - ) - for question in question_rows: - questions_by_schema.setdefault(question.schema_id, []).append(question) - - columns = _build_columns(schemas, questions_by_schema) - - if not schema_ids: - return WorkspaceProjection(columns=columns, rows=[], total_references=0) - - # Row universe: every distinct reference across the workspace's records (§3.1). - ref_query = select(V2Record.reference).where(V2Record.schema_id.in_(schema_ids)).distinct() - total_references = ( - await db.execute(select(func.count()).select_from(ref_query.subquery())) - ).scalar_one() - references = ( - (await db.execute(ref_query.order_by(V2Record.reference).offset(offset).limit(limit))) - .scalars() - .all() - ) - if not references: - return WorkspaceProjection(columns=columns, rows=[], total_references=total_references) - - record_rows = ( - ( - await db.execute( - select(V2Record).where( - V2Record.schema_id.in_(schema_ids), V2Record.reference.in_(references) - ) - ) - ) - .scalars() - .all() - ) - record_ids = [r.id for r in record_rows] - - sugg_rows: list[V2Suggestion] = [] - resp_rows: list[V2Response] = [] - if record_ids: - sugg_rows = ( - (await db.execute(select(V2Suggestion).where(V2Suggestion.record_id.in_(record_ids)))) - .scalars() - .all() - ) - # Submitted only — drafts must never reach the projection (§3.1). The - # latest-per-record pick happens in SQL (latest_responses window). - resp_rows = ( - ( - await db.execute( - select(V2Response).where( - V2Response.record_id.in_(record_ids), - V2Response.status == ResponseStatus.submitted, - ) - ) - ) - .scalars() - .all() - ) - - question_tuples: list[tuple] = [] - column_tuples: list[tuple] = [] - for schema in schemas: - for question in questions_by_schema.get(schema.id, []): - question_tuples.append( - (str(question.id), str(schema.id), schema.name, question.name, question.type.value) - ) - if question.type == QuestionType.table: - column_tuples.extend((str(question.id), sub) for sub in question.columns) - - inputs = { - "questions": question_tuples, - "question_columns": column_tuples, - "records": [(str(r.id), str(r.schema_id), r.reference, r.inserted_at) for r in record_rows], - "suggestions": [ - ( - str(s.record_id), - str(s.question_id), - json.dumps(s.value), - s.agent, - json.dumps(s.score) if s.score is not None else None, - ) - for s in sugg_rows - ], - "responses": [(str(r.record_id), json.dumps(r.values or {}), r.updated_at) for r in resp_rows], - } - long_rows = await to_thread.run_sync(_run_denormalization, inputs) - - # Regroup the ordered long format; SQL's ORDER BY reference matches the page order. - rows: list[WorkspaceProjectionRow] = [] - current: WorkspaceProjectionRow | None = None - for reference, row_index, column_name, value_json, source, record_id, agent, score_json in long_rows: - if current is None or current.reference != reference or current.row_index != row_index: - current = WorkspaceProjectionRow(reference=reference, row_index=row_index, cells={}) - rows.append(current) - if column_name is None: - continue # spine-only row: the reference exists but nothing resolved yet - current.cells[column_name] = WorkspaceProjectionCell( - value=json.loads(value_json), - source=source, - record_id=UUID(record_id), - agent=agent, - score=json.loads(score_json) if score_json is not None else None, - ) - return WorkspaceProjection(columns=columns, rows=rows, total_references=total_references) -``` - -Notes for the implementer: -- `build_reference_view`'s existing imports already cover most of this; merge rather than duplicate import lines. `AsyncSession`/`UUID` are already imported at the top of the module. -- `json_each` is DuckDB's SQLite-style table function (DuckDB ≥1.1) used laterally (`FROM t, json_each(t.col)`). If the installed version rejects the lateral column reference, replace those two spots with the zip-unnest pattern in the SELECT list: `unnest(from_json(col, '["json"]'))` paired with `unnest(generate_series(0, len(...) - 1))` — DuckDB zips parallel unnests. -- All JSON values cross the boundary as strings (`json.dumps` in, `CAST(... AS VARCHAR)` + `json.loads` out) — `score` round-trips `float | list[float]` intact this way. -- Timestamps insert as native `datetime`. If the ORM ever hands back tz-aware datetimes and DuckDB complains, switch the two DDL columns to `TIMESTAMPTZ`. + """Postgres serves batched raw slices; the DuckDB statement does all the semantics.""" +``` + +`build_workspace_view` — same batched-query skeleton as `build_reference_view`, generalized to the workspace: + +1. Fetch with ≤7 `AsyncSession` queries (the query-count test enforces this): schemas ordered by name → questions (`schema_id IN`) → distinct-reference count → the reference page (`ORDER BY reference OFFSET/LIMIT`) → all records for those references → suggestions and **submitted-only** responses for those record ids. Early-return an empty projection (with the columns manifest) when there are no schemas or no references. +2. Serialize the slices to plain tuples — UUIDs as `str`, JSON values via `json.dumps` (including `score`, so `float | list[float]` round-trips) — and run `_run_denormalization` via `await to_thread.run_sync(...)`: DuckDB is sync/CPU-bound; don't block the event loop. +3. Regroup the ordered long-format output into `WorkspaceProjectionRow`s (`json.loads` values/scores, `UUID(record_id)`). A row with a NULL column name is a spine-only row → empty `cells`. The SQL's `ORDER BY reference` matches the page order, so a single linear pass groups correctly. + +Input tables (all VARCHAR ids; JSON-typed value columns; TIMESTAMP for `inserted_at`/`updated_at`): +- `questions(question_id, schema_id, schema_name, question_name, qtype)` +- `question_columns(question_id, sub_column)` — table-question bindings pre-exploded in Python (avoids JSON-path gymnastics for arbitrary keys) +- `records(record_id, schema_id, reference, inserted_at)` +- `suggestions(record_id, question_id, value_json, agent, score_json)` +- `responses(record_id, values_json, updated_at)` + +`_DENORMALIZE_SQL` — one statement, one CTE per concern (behavior is fully pinned by the tests): +- `effective_records` — window dedup (`QUALIFY row_number() OVER ...`): latest `inserted_at` per (reference, schema_id) +- `latest_responses` — window dedup: latest `updated_at` per record, ANY user +- `response_cells` — `json_each` over the `{question_name: {"value": …}}` envelope +- `resolved` — questions × effective records with `COALESCE(response, suggestion)` + source/agent/score; (record, question) pairs with neither drop out +- `table_rows` — §3.4 normalization: wrap a bare dict via `json_array`, `json_each` arrays into a 0-based `row_idx`, keep only OBJECT rows +- `table_cells` — join `question_columns` and extract each bound sub-key (`'$."' || sub || '"'` quoting handles arbitrary key names); omit missing / JSON-null values +- `scalar_cells` — non-table resolved values, JSON-null omitted +- `spine` — per reference, `row_idx` from `generate_series(0, max table row_idx)`, min 1 row (independent stacking) +- final SELECT — spine LEFT JOIN (scalar cells repeated onto every `row_idx` UNION table cells at their own `row_idx`); `CAST` JSON columns to VARCHAR; `ORDER BY reference, row_idx, column_name` + +Gotchas: +- **NEVER `ATTACH` Postgres from DuckDB** — the test-fixture transaction is invisible to a second connection (see Interfaces). +- `json_each` used laterally (`FROM t, json_each(t.col)`) needs DuckDB ≥1.1; if the lateral column reference is rejected, use the zip-unnest pattern instead: `unnest(from_json(col, '["json"]'))` paired with `unnest(generate_series(...))` in the SELECT list. +- `build_reference_view`'s existing imports cover most needs — merge, don't duplicate; `AsyncSession`/`UUID` are already imported. +- If tz-aware datetimes upset the TIMESTAMP columns, switch them to `TIMESTAMPTZ`. - [ ] **Step 6: Run to verify pass** @@ -1076,64 +877,7 @@ Expected: FAIL — `getWorkspaceProjection is not a function`. - [ ] **Step 5: Implement the repository method** -In `v2/infrastructure/repositories/ProjectionRepository.ts` add (keeping the existing `getProjection` untouched): - -```ts -import { - type ProjectionColumn, - type ProjectionGridCell, - type ProjectionGridRow, -} from "~/v2/domain/entities/projection/WorkspaceProjection"; - -type BackendWorkspaceProjection = components["schemas"]["WorkspaceProjection"]; - -export interface WorkspaceProjectionPageDto { - columns: ProjectionColumn[]; - rows: ProjectionGridRow[]; - totalReferences: number; -} -``` - -and inside the class: - -```ts - async getWorkspaceProjection( - workspaceId: string, - offset = 0, - limit = 50 - ): Promise { - const { data } = await this.axios.get("/v2/projection", { - params: { workspace_id: workspaceId, offset, limit }, - }); - return { - totalReferences: data.total_references, - columns: data.columns.map((c) => ({ - name: c.name, - schemaId: c.schema_id, - schemaName: c.schema_name, - questionName: c.question_name, - subColumn: c.sub_column ?? null, - dtype: c.dtype, - })), - rows: data.rows.map((r) => ({ - reference: r.reference, - rowIndex: r.row_index, - cells: Object.fromEntries( - Object.entries(r.cells).map(([name, cell]) => [ - name, - { - value: cell.value ?? null, - source: cell.source, - recordId: cell.record_id, - agent: cell.agent ?? null, - score: cell.score ?? null, - } satisfies ProjectionGridCell, - ]) - ), - })), - }; - } -``` +In `v2/infrastructure/repositories/ProjectionRepository.ts` (keep the existing `getProjection` untouched): export `WorkspaceProjectionPageDto` (shape in Interfaces) and add `getWorkspaceProjection(workspaceId, offset = 0, limit = 50)` calling `GET "/v2/projection"` with `params: { workspace_id, offset, limit }`, typed against `components["schemas"]["WorkspaceProjection"]` from the generated file, mapping snake_case → camelCase. Follow the existing method's style; the test pins the exact mapping, including `?? null` for the optional cell fields. - [ ] **Step 6: Run to verify pass** @@ -1256,51 +1000,11 @@ Expected: FAIL — module not found. - [ ] **Step 3: Implement** -Create `v2/domain/entities/projection/grid-adapter.ts`: - -```ts -import { type ProjectionGridCell, type WorkspaceProjection } from "./WorkspaceProjection"; - -export const REFERENCE_COLUMN = "reference"; - -// Flat denormalized records for `client.table(data)`. Every manifest column is present -// on every row (null when absent) so Perspective infers a stable schema. -export const toPerspectiveData = (projection: WorkspaceProjection): Record[] => - projection.rows.map((row) => { - const flat: Record = { [REFERENCE_COLUMN]: row.reference }; - for (const column of projection.columns) { - flat[column.name] = row.cells[column.name]?.value ?? null; - } - return flat; - }); - -// Click plumbing (spec §3.3): rows load in projection order and the grid is static -// (no sort/filter), so a datagrid row index maps 1:1 onto projection.rows. -export const cellAt = ( - projection: WorkspaceProjection, - rowIndex: number, - columnName: string -): ProjectionGridCell | null => projection.rows[rowIndex]?.cells[columnName] ?? null; - -// Row-banding by reference (spec §3.1) — Perspective is flat-columnar and cannot merge -// cells, so visual grouping is alternating bands per reference block. -export const bandParity = (projection: WorkspaceProjection): number[] => { - let parity = 0; - let previous: string | null = null; - return projection.rows.map((row) => { - if (previous !== null && row.reference !== previous) parity ^= 1; - previous = row.reference; - return parity; - }); -}; - -// Future annotation URL contract (spec §3.3): schema id occupies the dataset slot. -// Guarded OFF until annotation-mode resolves v2 schema ids (ledger §5). -export const ANNOTATION_CELL_LINKS_ENABLED = false; - -export const buildAnnotationUrl = (schemaId: string, reference: string): string => - `/dataset/${schemaId}/annotation-mode?_search=${encodeURIComponent(reference)}`; -``` +Create `v2/domain/entities/projection/grid-adapter.ts` with the five exports listed in Interfaces — all pure, behavior fully pinned by the tests. Design intent to preserve in comments: +- `toPerspectiveData` emits EVERY manifest column on every row (`null` when absent) so Perspective infers a stable schema, with `reference` as the first key. +- `cellAt` relies on the static-grid invariant (no sort/filter): a datagrid row index maps 1:1 onto `projection.rows`. +- `bandParity` flips 0↔1 when `reference` changes — Perspective can't merge cells, so banding is the §3.1 reference-grouping affordance. +- `buildAnnotationUrl` percent-encodes the reference; `ANNOTATION_CELL_LINKS_ENABLED = false` guards navigation until annotation-mode resolves v2 schema ids (ledger §5). - [ ] **Step 4: Run to verify pass** @@ -1421,45 +1125,7 @@ Expected: FAIL — module not found. - [ ] **Step 4: Implement the use-case** -Create `v2/domain/usecases/get-workspace-projection-use-case.ts`: - -```ts -import { WorkspaceProjection } from "../entities/projection/WorkspaceProjection"; -import { ProjectionRepository } from "~/v2/infrastructure/repositories/ProjectionRepository"; -import { type useExtractions } from "~/v2/infrastructure/storage/ExtractionsStorage"; - -// Server caps limit at 100; the client loads all pages into ONE Perspective table -// (spec §3.3 data path — Arrow IPC streaming is a recorded follow-up, §5). -export const PROJECTION_PAGE_SIZE = 100; - -export class GetWorkspaceProjectionUseCase { - constructor( - private readonly projectionRepository: ProjectionRepository, - // ts-injecty resolves the hook by calling it; the injected value is the store object. - private readonly extractionsStorage: ReturnType - ) {} - - async execute(workspaceId: string): Promise { - const first = await this.projectionRepository.getWorkspaceProjection( - workspaceId, - 0, - PROJECTION_PAGE_SIZE - ); - const rows = [...first.rows]; - for (let offset = PROJECTION_PAGE_SIZE; offset < first.totalReferences; offset += PROJECTION_PAGE_SIZE) { - const page = await this.projectionRepository.getWorkspaceProjection( - workspaceId, - offset, - PROJECTION_PAGE_SIZE - ); - rows.push(...page.rows); - } - const projection = new WorkspaceProjection(first.columns, rows, first.totalReferences); - this.extractionsStorage.saveProjection(projection); - return projection; - } -} -``` +Create `v2/domain/usecases/get-workspace-projection-use-case.ts` exporting `PROJECTION_PAGE_SIZE = 100` (the server's `limit` cap) and `GetWorkspaceProjectionUseCase` with constructor `(projectionRepository: ProjectionRepository, extractionsStorage: ReturnType)` — ts-injecty resolves the hook by calling it, so the injected value is the store object (same contract as `GetReferenceReviewUseCase` today). `execute(workspaceId)` fetches offset 0, keeps paging by `PROJECTION_PAGE_SIZE` until `totalReferences` is covered, concatenates all rows under the first page's columns/total into ONE `WorkspaceProjection`, saves it via `saveProjection`, and returns it. The test pins the call sequence; comment why it loads everything (one Perspective table — Arrow IPC streaming is the recorded §5 follow-up). - [ ] **Step 5: Run to verify pass** @@ -1737,128 +1403,15 @@ Expected: FAIL — component file not found. - [ ] **Step 3: Implement the component** -Create `components/v2/extractions/ExtractionsGrid.vue`: - -```vue - - - - - -``` - -Note: if `restore({ plugin: "Datagrid" })` names the plugin differently in 4.5.2, the accepted values are listed by `viewer.restore` typings in `node_modules/@perspective-dev/viewer/dist/esm/perspective-viewer.d.ts` — the datagrid plugin registers as `"Datagrid"`. The `:deep()` selectors only apply if the datagrid renders in light DOM (it is slotted into the viewer); if banding doesn't show in Task 12's e2e run, move the two rules to an unscoped ` -``` - -(`useEnsureWorkspaces` and `useV2Breadcrumbs` are Nuxt auto-imported composables — match how `pages/schemas/index.vue` references them; if that file imports them explicitly, do the same here.) +Create `pages/extractions/index.vue` mirroring `pages/schemas/index.vue` exactly (same `InternalPage`/`AppHeader` shell, same `setup()` style, `useEnsureWorkspaces` + `useV2Breadcrumbs` wiring, `onBeforeMount` → `await ensureWorkspaces()` then `await load()`), with two page-specific behaviors: +- Read `route.query.workspace_id` (string only) and pass it to `useExtractionsViewModel` as the override. +- Render this state cascade in `#page-content` under an `$t("extractions.title")` heading: no workspace → `noWorkspace`; failed → `loadError`; loading → `loading`; loaded with zero rows → `empty`; otherwise ``. - [ ] **Step 7: Full check + smoke** @@ -2127,28 +1563,7 @@ git commit -m "feat(v2-ui): /extractions page with Perspective grid and workspac - [ ] **Step 1: Extract the kept types** -Create `v2/domain/entities/review/ReviewCell.ts` (moved verbatim from `ReferenceReview.ts`): - -```ts -import { Question } from "../question/Question"; - -export interface Provenance { - agent: string | null; - score: number | null; - suggestedValue: unknown; -} - -export class ReviewCell { - constructor( - public readonly question: Question, - public readonly value: unknown, - public readonly source: "response" | "suggestion" | null, - public readonly provenance: Provenance | null, - // The question binds a column absent from this record's pinned version cache (§17.3). - public readonly notApplicable: boolean - ) {} -} -``` +Create `v2/domain/entities/review/ReviewCell.ts` and move the `Provenance` and `ReviewCell` declarations into it VERBATIM from `ReferenceReview.ts` (they depend only on `Question`), including their comments. Do NOT move `ReferenceReview`/`ReviewRecord`/`ContextField`/`OrphanedValue` — those die with the page. - [ ] **Step 2: Repoint the kept importers** @@ -2222,74 +1637,12 @@ git commit -m "refactor(v2-ui): delete reference-review page — review derives - [ ] **Step 1: Extend the seed script** -In `e2e/v2/seed/seed_v2_e2e.py`: - -Top-of-file constants (next to `SCHEMA_NAME`): - -```python -EMPTY_SCHEMA_NAME = "e2e_v2_empty" - -EMPTY_BODY = pa.DataFrameSchema( - columns={"notes": pa.Column(pa.String, nullable=True)} -).to_json() -``` - -Extend the schema-recreate loop (currently deletes only `SCHEMA_NAME`): - -```python - for schema_item in schemas: - if schema_item["name"] in (SCHEMA_NAME, EMPTY_SCHEMA_NAME): - client.delete(f"/api/v2/schemas/{schema_item['id']}").raise_for_status() -``` - -After the existing suggestion `client.put(...suggestions...)` block, add: +In `e2e/v2/seed/seed_v2_e2e.py`, mirroring the script's OWN existing client calls (schema create → version → questions → record → suggestion; copy their exact endpoint/payload shapes rather than inventing new ones): -```python - # Response-beats-suggestion on `label`: competing suggestion + submitted response. - client.put( - f"/api/v2/records/{record['id']}/suggestions", - json={ - "question_id": questions["label"]["id"], - "value": "intervention", - "score": 0.55, - "agent": "e2e-seeder", - }, - ).raise_for_status() - client.put( - f"/api/v2/records/{record['id']}/responses", - json={"values": {"label": {"value": "control"}}, "status": "submitted"}, - ).raise_for_status() - - # Second schema with zero records: the grid doubles as a coverage map (§3.1). - empty_schema = ( - client.post( - "/api/v2/schemas", - json={"name": EMPTY_SCHEMA_NAME, "workspace_id": workspace["id"]}, - ) - .raise_for_status() - .json() - ) - client.post( - f"/api/v2/schemas/{empty_schema['id']}/versions", json={"body": EMPTY_BODY} - ).raise_for_status() - client.post( - f"/api/v2/schemas/{empty_schema['id']}/questions", - json={ - "name": "notes", - "title": "Notes", - "type": "text", - "columns": ["notes"], - "settings": {}, - "required": False, - }, - ).raise_for_status() -``` - -And extend the output dict: - -```python - "emptySchemaName": EMPTY_SCHEMA_NAME, -``` +1. Add an `EMPTY_SCHEMA_NAME = "e2e_v2_empty"` constant and include it in the delete-and-recreate loop that currently handles only `SCHEMA_NAME`. +2. On the seeded record's `label` question, create a competing suggestion (`value="intervention"`, `agent="e2e-seeder"`, any score) AND a submitted response (`values={"label": {"value": "control"}}`, `status="submitted"` — the `PUT /records/{id}/responses` shape the deleted review specs used) so the grid proves response-beats-suggestion. +3. Create the second schema `EMPTY_SCHEMA_NAME` with one version, one text question `notes` (bound `columns=["notes"]`), and ZERO records — the grid doubles as a coverage map (§3.1). +4. Add `"emptySchemaName": EMPTY_SCHEMA_NAME` to the seed-output dict. In `e2e/v2/fixtures.ts`, add `emptySchemaName: string;` to the `SeedOutput` type. From 37b77c757e0c592da6c759307eddf59396f75446 Mon Sep 17 00:00:00 2001 From: JonnyTran Date: Tue, 21 Jul 2026 16:41:34 -0700 Subject: [PATCH 20/21] fix(server): narrow table score exemption to scalar; tighten review tests MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Addresses roborev job #228 (auto-review of 2a1ac44f7). Medium — the table score exemption was total, accepting arbitrary list scores. That contradicts its own rationale (a table suggestion's score is whole-suggestion confidence) and is not inert: the fan-out repeats the whole list onto every cell, surfacing a multi-value score no consumer can read. Narrowed to `score is None or isinstance(score, (int, float))`; a per-row score is a distinct future feature needing indexed fan-out. Flipped the two list-accepting tests to assert rejection; mutation-verified they fail without the guard. Low — the two non-table negative tests paired type=multi_label_selection with LABEL_SETTINGS (settings type label_selection), a combination the API cannot persist that only worked because both settings models expose `.options`. Added "no" to MULTI_LABEL_SETTINGS and use it, so the type/settings pair matches what QuestionSettingsValidator enforces. Low — test_non_ascii_names_and_bindings_resolve keyed its response envelope on ASCII "pays", so the responses-side ensure_ascii=False was untested. Renamed the scalar question to "país" and key the envelope with it, exercising the response-envelope join (rc.question_name = q.question_name) against a non-ASCII key. Server 138 passed; ruff clean. --- .../extralit_server/validators/v2/values.py | 13 ++++--- .../contexts/v2/test_workspace_projection.py | 9 +++-- .../tests/unit/validators/v2/test_values.py | 35 +++++++++++-------- 3 files changed, 34 insertions(+), 23 deletions(-) diff --git a/extralit-server/src/extralit_server/validators/v2/values.py b/extralit-server/src/extralit_server/validators/v2/values.py index 6c74d94b9..8b5e3348f 100644 --- a/extralit-server/src/extralit_server/validators/v2/values.py +++ b/extralit-server/src/extralit_server/validators/v2/values.py @@ -71,11 +71,14 @@ def validate(cls, value, score, *, type: QuestionType, settings: dict, columns: @staticmethod def _validate_score(value, score, *, type: QuestionType) -> None: if type == QuestionType.table: - # The cardinality rules below assume a list value is a list of *answer choices* - # (multi_label_selection, ranking), where one score per choice is meaningful. - # A table value's list is N *rows* (spec §3.4), and a suggestion's score applies - # to the suggestion as a whole — the projection fan-out repeats it onto every - # fanned-out cell. Row count and score count are unrelated, so skip the checks. + # A table value's list is N *rows* (spec §3.4), not N answer choices, so the + # answer-choice cardinality rules below don't apply. A suggestion's score is + # whole-suggestion confidence — a scalar or None — which the projection fan-out + # repeats onto every fanned-out cell. A per-row score list would be a distinct + # future feature (needing indexed fan-out, not whole-list repetition); reject it + # now rather than surface an uninterpretable multi-value score in the grid. + if score is not None and not isinstance(score, (int, float)): + raise UnprocessableEntityError("a table question score must be a single number or null") return if not isinstance(value, list) and isinstance(score, list): raise UnprocessableEntityError("a list of scores is not allowed for a single-value suggestion") diff --git a/extralit-server/tests/integration/contexts/v2/test_workspace_projection.py b/extralit-server/tests/integration/contexts/v2/test_workspace_projection.py index 7d403b088..a0c2646d7 100644 --- a/extralit-server/tests/integration/contexts/v2/test_workspace_projection.py +++ b/extralit-server/tests/integration/contexts/v2/test_workspace_projection.py @@ -280,18 +280,21 @@ async def test_non_ascii_names_and_bindings_resolve(db): # serialization) — a mismatch would silently omit the cell rather than error. workspace = await WorkspaceFactory.create() schema, version = await _make_schema(workspace, "Résumé") - await _add_question(schema, "pays") + # `país` is non-ASCII so it exercises the response-envelope join (rc.question_name = + # q.question_name) against a key DuckDB parsed from values_json — the half of the + # ensure_ascii=False change on the responses serialization that an ASCII name would miss. + await _add_question(schema, "país") table_q = await _add_question(schema, "résultats", qtype=QuestionType.table, columns=["café", "日本語"]) rec = await V2RecordFactory.create(version=version, reference="10.1/a") user = await UserFactory.create() await V2SuggestionFactory.create(record=rec, question=table_q, value=[{"café": "noir", "日本語": "はい"}]) await V2ResponseFactory.create( - record=rec, user=user, status=ResponseStatus.submitted, values={"pays": {"value": "Côte d'Ivoire"}} + record=rec, user=user, status=ResponseStatus.submitted, values={"país": {"value": "Côte d'Ivoire"}} ) view = await projection_ctx.build_workspace_view(db, workspace_id=workspace.id, offset=0, limit=50) cells = view.rows[0].cells - assert cells["Résumé.pays"].value == "Côte d'Ivoire" + assert cells["Résumé.país"].value == "Côte d'Ivoire" assert cells["Résumé.résultats.café"].value == "noir" assert cells["Résumé.résultats.日本語"].value == "はい" diff --git a/extralit-server/tests/unit/validators/v2/test_values.py b/extralit-server/tests/unit/validators/v2/test_values.py index 2c85c493e..34102148b 100644 --- a/extralit-server/tests/unit/validators/v2/test_values.py +++ b/extralit-server/tests/unit/validators/v2/test_values.py @@ -48,7 +48,7 @@ def test_unknown_question_type_fails_closed(): MULTI_LABEL_SETTINGS = { "type": "multi_label_selection", - "options": [{"value": "yes", "text": "Yes"}], + "options": [{"value": "yes", "text": "Yes"}, {"value": "no", "text": "No"}], } @@ -99,27 +99,32 @@ def test_table_suggestion_allows_single_score_for_multi_row_value(): ) -def test_table_suggestion_allows_score_list_not_matching_row_count(): - V2SuggestionValidator.validate( - [{"value": "12%"}, {"value": "8%"}, {"value": "3%"}], - [0.9, 0.1], - type=QuestionType.table, - settings=TABLE_SETTINGS, - columns=["value"], - ) +def test_table_suggestion_rejects_score_list(): + # The exemption is scalar-only: a table score is whole-suggestion confidence, so a list + # (which the fan-out would repeat verbatim onto every cell) is uninterpretable, not per-row. + with pytest.raises(UnprocessableEntityError, match="single number or null"): + V2SuggestionValidator.validate( + [{"value": "12%"}, {"value": "8%"}, {"value": "3%"}], + [0.9, 0.1], + type=QuestionType.table, + settings=TABLE_SETTINGS, + columns=["value"], + ) -def test_table_suggestion_allows_score_list_for_bare_dict_value(): - V2SuggestionValidator.validate( - {"value": "12%"}, [0.9, 0.1], type=QuestionType.table, settings=TABLE_SETTINGS, columns=["value"] - ) +def test_table_suggestion_rejects_score_list_for_bare_dict_value(): + with pytest.raises(UnprocessableEntityError, match="single number or null"): + V2SuggestionValidator.validate( + {"value": "12%"}, [0.9, 0.1], type=QuestionType.table, settings=TABLE_SETTINGS, columns=["value"] + ) def test_non_table_suggestion_still_rejects_single_score_for_list_value(): # The exemption above must be scoped to table only — these rules still bind elsewhere. + # Use MULTI_LABEL_SETTINGS so the settings `type` matches the question type the API enforces. with pytest.raises(UnprocessableEntityError, match="a single score is not allowed"): V2SuggestionValidator.validate( - ["yes", "no"], 0.9, type=QuestionType.multi_label_selection, settings=LABEL_SETTINGS, columns=["q"] + ["yes", "no"], 0.9, type=QuestionType.multi_label_selection, settings=MULTI_LABEL_SETTINGS, columns=["q"] ) @@ -129,6 +134,6 @@ def test_non_table_suggestion_still_rejects_mismatched_score_count(): ["yes", "no"], [0.9], type=QuestionType.multi_label_selection, - settings=LABEL_SETTINGS, + settings=MULTI_LABEL_SETTINGS, columns=["q"], ) From 0ee7999eb924a3f2ef872543de76d65568d325e6 Mon Sep 17 00:00:00 2001 From: JonnyTran Date: Tue, 21 Jul 2026 16:49:12 -0700 Subject: [PATCH 21/21] fix(v2-ui): regenerate OpenAPI snapshot under py3.12 to match CI (422 phrase) The v2 OpenAPI drift gate dumps the schema under CI's Python (<=3.12), where HTTPStatus(422).phrase is 'Unprocessable Entity'. The committed snapshot was generated under Python 3.13, which renamed the phrase to 'Unprocessable Content', so the full-file diff -u failed across all 28 occurrences. Regenerated openapi.json + v2-api.ts under Python 3.12; no schema/type changes beyond the phrase. --- .../v2/infrastructure/api/generated/v2-api.ts | 56 +++++++++---------- .../v2/infrastructure/api/openapi.json | 56 +++++++++---------- 2 files changed, 56 insertions(+), 56 deletions(-) diff --git a/extralit-frontend/v2/infrastructure/api/generated/v2-api.ts b/extralit-frontend/v2/infrastructure/api/generated/v2-api.ts index ca9a055a6..f826f04d2 100644 --- a/extralit-frontend/v2/infrastructure/api/generated/v2-api.ts +++ b/extralit-frontend/v2/infrastructure/api/generated/v2-api.ts @@ -1006,7 +1006,7 @@ export interface operations { "application/json": unknown; }; }; - /** @description Unprocessable Content */ + /** @description Unprocessable Entity */ 422: { headers: { [name: string]: unknown; @@ -1142,7 +1142,7 @@ export interface operations { "application/json": unknown; }; }; - /** @description Unprocessable Content */ + /** @description Unprocessable Entity */ 422: { headers: { [name: string]: unknown; @@ -1275,7 +1275,7 @@ export interface operations { "application/json": unknown; }; }; - /** @description Unprocessable Content */ + /** @description Unprocessable Entity */ 422: { headers: { [name: string]: unknown; @@ -1412,7 +1412,7 @@ export interface operations { "application/json": unknown; }; }; - /** @description Unprocessable Content */ + /** @description Unprocessable Entity */ 422: { headers: { [name: string]: unknown; @@ -1543,7 +1543,7 @@ export interface operations { "application/json": unknown; }; }; - /** @description Unprocessable Content */ + /** @description Unprocessable Entity */ 422: { headers: { [name: string]: unknown; @@ -1676,7 +1676,7 @@ export interface operations { "application/json": unknown; }; }; - /** @description Unprocessable Content */ + /** @description Unprocessable Entity */ 422: { headers: { [name: string]: unknown; @@ -1813,7 +1813,7 @@ export interface operations { "application/json": unknown; }; }; - /** @description Unprocessable Content */ + /** @description Unprocessable Entity */ 422: { headers: { [name: string]: unknown; @@ -1946,7 +1946,7 @@ export interface operations { "application/json": unknown; }; }; - /** @description Unprocessable Content */ + /** @description Unprocessable Entity */ 422: { headers: { [name: string]: unknown; @@ -2083,7 +2083,7 @@ export interface operations { "application/json": unknown; }; }; - /** @description Unprocessable Content */ + /** @description Unprocessable Entity */ 422: { headers: { [name: string]: unknown; @@ -2219,7 +2219,7 @@ export interface operations { "application/json": unknown; }; }; - /** @description Unprocessable Content */ + /** @description Unprocessable Entity */ 422: { headers: { [name: string]: unknown; @@ -2353,7 +2353,7 @@ export interface operations { "application/json": unknown; }; }; - /** @description Unprocessable Content */ + /** @description Unprocessable Entity */ 422: { headers: { [name: string]: unknown; @@ -2488,7 +2488,7 @@ export interface operations { "application/json": unknown; }; }; - /** @description Unprocessable Content */ + /** @description Unprocessable Entity */ 422: { headers: { [name: string]: unknown; @@ -2621,7 +2621,7 @@ export interface operations { "application/json": unknown; }; }; - /** @description Unprocessable Content */ + /** @description Unprocessable Entity */ 422: { headers: { [name: string]: unknown; @@ -2758,7 +2758,7 @@ export interface operations { "application/json": unknown; }; }; - /** @description Unprocessable Content */ + /** @description Unprocessable Entity */ 422: { headers: { [name: string]: unknown; @@ -2891,7 +2891,7 @@ export interface operations { "application/json": unknown; }; }; - /** @description Unprocessable Content */ + /** @description Unprocessable Entity */ 422: { headers: { [name: string]: unknown; @@ -3024,7 +3024,7 @@ export interface operations { "application/json": unknown; }; }; - /** @description Unprocessable Content */ + /** @description Unprocessable Entity */ 422: { headers: { [name: string]: unknown; @@ -3157,7 +3157,7 @@ export interface operations { "application/json": unknown; }; }; - /** @description Unprocessable Content */ + /** @description Unprocessable Entity */ 422: { headers: { [name: string]: unknown; @@ -3294,7 +3294,7 @@ export interface operations { "application/json": unknown; }; }; - /** @description Unprocessable Content */ + /** @description Unprocessable Entity */ 422: { headers: { [name: string]: unknown; @@ -3432,7 +3432,7 @@ export interface operations { "application/json": unknown; }; }; - /** @description Unprocessable Content */ + /** @description Unprocessable Entity */ 422: { headers: { [name: string]: unknown; @@ -3566,7 +3566,7 @@ export interface operations { "application/json": unknown; }; }; - /** @description Unprocessable Content */ + /** @description Unprocessable Entity */ 422: { headers: { [name: string]: unknown; @@ -3703,7 +3703,7 @@ export interface operations { "application/json": unknown; }; }; - /** @description Unprocessable Content */ + /** @description Unprocessable Entity */ 422: { headers: { [name: string]: unknown; @@ -3840,7 +3840,7 @@ export interface operations { "application/json": unknown; }; }; - /** @description Unprocessable Content */ + /** @description Unprocessable Entity */ 422: { headers: { [name: string]: unknown; @@ -3973,7 +3973,7 @@ export interface operations { "application/json": unknown; }; }; - /** @description Unprocessable Content */ + /** @description Unprocessable Entity */ 422: { headers: { [name: string]: unknown; @@ -4110,7 +4110,7 @@ export interface operations { "application/json": unknown; }; }; - /** @description Unprocessable Content */ + /** @description Unprocessable Entity */ 422: { headers: { [name: string]: unknown; @@ -4244,7 +4244,7 @@ export interface operations { "application/json": unknown; }; }; - /** @description Unprocessable Content */ + /** @description Unprocessable Entity */ 422: { headers: { [name: string]: unknown; @@ -4379,7 +4379,7 @@ export interface operations { "application/json": unknown; }; }; - /** @description Unprocessable Content */ + /** @description Unprocessable Entity */ 422: { headers: { [name: string]: unknown; @@ -4514,7 +4514,7 @@ export interface operations { "application/json": unknown; }; }; - /** @description Unprocessable Content */ + /** @description Unprocessable Entity */ 422: { headers: { [name: string]: unknown; @@ -4649,7 +4649,7 @@ export interface operations { "application/json": unknown; }; }; - /** @description Unprocessable Content */ + /** @description Unprocessable Entity */ 422: { headers: { [name: string]: unknown; diff --git a/extralit-frontend/v2/infrastructure/api/openapi.json b/extralit-frontend/v2/infrastructure/api/openapi.json index e75ed373e..4286722d3 100644 --- a/extralit-frontend/v2/infrastructure/api/openapi.json +++ b/extralit-frontend/v2/infrastructure/api/openapi.json @@ -1537,7 +1537,7 @@ } } }, - "description": "Unprocessable Content" + "description": "Unprocessable Entity" }, "500": { "content": { @@ -1676,7 +1676,7 @@ } } }, - "description": "Unprocessable Content" + "description": "Unprocessable Entity" }, "500": { "content": { @@ -1797,7 +1797,7 @@ } } }, - "description": "Unprocessable Content" + "description": "Unprocessable Entity" }, "500": { "content": { @@ -1923,7 +1923,7 @@ } } }, - "description": "Unprocessable Content" + "description": "Unprocessable Entity" }, "500": { "content": { @@ -2059,7 +2059,7 @@ } } }, - "description": "Unprocessable Content" + "description": "Unprocessable Entity" }, "500": { "content": { @@ -2195,7 +2195,7 @@ } } }, - "description": "Unprocessable Content" + "description": "Unprocessable Entity" }, "500": { "content": { @@ -2331,7 +2331,7 @@ } } }, - "description": "Unprocessable Content" + "description": "Unprocessable Entity" }, "500": { "content": { @@ -2459,7 +2459,7 @@ } } }, - "description": "Unprocessable Content" + "description": "Unprocessable Entity" }, "500": { "content": { @@ -2595,7 +2595,7 @@ } } }, - "description": "Unprocessable Content" + "description": "Unprocessable Entity" }, "500": { "content": { @@ -2735,7 +2735,7 @@ } } }, - "description": "Unprocessable Content" + "description": "Unprocessable Entity" }, "500": { "content": { @@ -2865,7 +2865,7 @@ } } }, - "description": "Unprocessable Content" + "description": "Unprocessable Entity" }, "500": { "content": { @@ -2989,7 +2989,7 @@ } } }, - "description": "Unprocessable Content" + "description": "Unprocessable Entity" }, "500": { "content": { @@ -3117,7 +3117,7 @@ } } }, - "description": "Unprocessable Content" + "description": "Unprocessable Entity" }, "500": { "content": { @@ -3243,7 +3243,7 @@ } } }, - "description": "Unprocessable Content" + "description": "Unprocessable Entity" }, "500": { "content": { @@ -3379,7 +3379,7 @@ } } }, - "description": "Unprocessable Content" + "description": "Unprocessable Entity" }, "500": { "content": { @@ -3511,7 +3511,7 @@ } } }, - "description": "Unprocessable Content" + "description": "Unprocessable Entity" }, "500": { "content": { @@ -3639,7 +3639,7 @@ } } }, - "description": "Unprocessable Content" + "description": "Unprocessable Entity" }, "500": { "content": { @@ -3775,7 +3775,7 @@ } } }, - "description": "Unprocessable Content" + "description": "Unprocessable Entity" }, "500": { "content": { @@ -3907,7 +3907,7 @@ } } }, - "description": "Unprocessable Content" + "description": "Unprocessable Entity" }, "500": { "content": { @@ -4088,7 +4088,7 @@ } } }, - "description": "Unprocessable Content" + "description": "Unprocessable Entity" }, "500": { "content": { @@ -4226,7 +4226,7 @@ } } }, - "description": "Unprocessable Content" + "description": "Unprocessable Entity" }, "500": { "content": { @@ -4365,7 +4365,7 @@ } } }, - "description": "Unprocessable Content" + "description": "Unprocessable Entity" }, "500": { "content": { @@ -4497,7 +4497,7 @@ } } }, - "description": "Unprocessable Content" + "description": "Unprocessable Entity" }, "500": { "content": { @@ -4633,7 +4633,7 @@ } } }, - "description": "Unprocessable Content" + "description": "Unprocessable Entity" }, "500": { "content": { @@ -4770,7 +4770,7 @@ } } }, - "description": "Unprocessable Content" + "description": "Unprocessable Entity" }, "500": { "content": { @@ -4903,7 +4903,7 @@ } } }, - "description": "Unprocessable Content" + "description": "Unprocessable Entity" }, "500": { "content": { @@ -5029,7 +5029,7 @@ } } }, - "description": "Unprocessable Content" + "description": "Unprocessable Entity" }, "500": { "content": { @@ -5148,7 +5148,7 @@ } } }, - "description": "Unprocessable Content" + "description": "Unprocessable Entity" }, "500": { "content": {