feat(sdk): v2 SDK vertical slice — generated DTOs, async transport, resources, agentic CLI

docs/superpowers/specs/2026-07-13-sdk-v2-redesign-design.md

@@ -0,0 +1,223 @@
+# Python SDK v2 — schema-centric client, agentic CLI, async performance
+
+**Date:** 2026-07-13
+**Status:** Approved design, pending implementation plan
+**Companions:** `2026-06-27-schema-centric-data-model-design.md` (server model),
+`2026-07-09-v2-frontend-vertical-slice-design.md` (frontend precedent this design mirrors)
+
+## Problem
+
+The backend grew a second data model (`/api/v2`: schema-centric, versioned,
+projection-based) alongside the Argilla-lineage one (`/api/v1`). The Python SDK
+(`extralit/`, v0.6.1) is hardcoded to `/api/v1`: sync-only `httpx.Client`, Argilla
+entities (Dataset → Record → Question → Response), and a human-oriented Rich CLI.
+It cannot drive the v2 extraction loop at all, and three of its properties block the
+intended consumers:
+
+1. **Wrong model.** No client for schemas/versions, version-pinned records,
+   column-bound questions, reference grouping, or projections.
+2. **Not agentic.** CLI output is Rich tables only — unparseable by agents; no
+   stable JSON, no meaningful exit-code contract, heavy imports make every
+   short-lived invocation slow.
+3. **Sequential I/O.** All HTTP is blocking and serial; pushing large record sets
+   or fanning out per-record reads requires user-written loops.
+
+## Decision summary
+
+| Decision | Choice |
+|---|---|
+| Coexistence | Parallel `extralit.v2` package; v1 untouched; one wheel |
+| Contract | Committed `openapi-dump` snapshot → generated Pydantic DTOs (Approach A) |
+| Transport | Async-native (`httpx.AsyncClient`) + sync facade |
+| CLI | Top-level verbs (`extralit schemas`, `extralit records`, …) — v2 owns the CLI surface; JSON-first, lazy-imported |
+| Primary workflow | LLM extraction pipeline (publish schema → bulk-upsert → suggestions → projections) |
+| Scope | Vertical slice (below); responses-write, exports, admin verbs deferred |
+
+Alternatives rejected: **evolve-in-place** (recreates the name-collision /
+entity-bending trap the frontend explicitly avoided — a v2 `Record` is not a v1
+`Record`); **fully generated client** via `openapi-python-client` (generated
+ergonomics fight the agentic-CLI and pipeline goals on every regen); **hand-written
+wire models** (silent drift against a quirky contract we already have a dump tool for).
+
+## 1. Package architecture
+
+```
+src/extralit/v2/
+├── __init__.py           # exports AsyncClient, Client, domain classes
+├── _api/
+│   ├── openapi.json      # committed snapshot (server `openapi-dump`)
+│   ├── _generated.py     # datamodel-code-generator → Pydantic v2 DTOs
+│   ├── _transport.py     # httpx.AsyncClient wrapper: bearer auth + refresh, retries
+│   └── _errors.py        # error hierarchy + 422 normalizer
+├── _sync.py              # sync facade (background-thread event-loop portal)
+├── models/               # hand-written domain classes (Schema, SchemaVersion, Record, …)
+├── resources/            # Schemas, Records, Questions, Suggestions, Projections
+└── cli/                  # top-level CLI verbs (lazy-registered into cli/app.py)
+```
+
+Python namespacing removes the frontend's need for `V2`-prefixed names:
+`extralit.v2.Record` cannot collide with `extralit.Record`. Entry points are
+explicit — `from extralit.v2 import Client` (sync) / `AsyncClient` — never re-exported
+from the top-level `extralit` namespace.
+
+**Import wall (reuse-don't-fork, with a hard boundary).** `extralit.v2` imports
+nothing from v1 modules, with one documented exception: a shared credentials helper
+(env vars + `~/.extralit/credentials.json`), which survives v1 retirement anyway.
+v1 never imports v2. The boundary is grep-checkable and gated in CI. Phase 6
+retirement deletes v1 packages mechanically. `cli/app.py` is the composition root
+(like the frontend's `plugins/3.di.ts`): it may import `extralit.v2.cli` to register
+commands without counting as a boundary violation.
+
+## 2. Contract layer
+
+`_api/openapi.json` is a checked-in dump of the server's v2 OpenAPI schema, produced
+by the existing server CLI (`uv run python -m extralit_server openapi-dump`).
+`datamodel-code-generator` (dev dependency, `--output-model-type pydantic_v2.BaseModel`)
+turns it into `_generated.py`. Resources type every wire payload from generated DTOs;
+hand-written shapes never touch the wire.
+
+Two CI gates, mirroring the frontend's:
+
+- **No-drift:** regenerate from the committed snapshot and diff; any change fails.
+- **Snapshot-vs-server:** the committed snapshot must match a fresh `openapi-dump`
+  from the server tree at the same monorepo revision.
+
+## 3. Transport & performance
+
+- **Async-native.** One `httpx.AsyncClient` per client instance: pooled connections,
+  transport `retries=5`, 60s default timeout — matching v1's proven settings.
+- **Auth.** v2 routes accept both backends (verified: `security/authentication/
+  provider.py` registers `APIKeyAuthenticationBackend` and
+  `BearerTokenAuthenticationBackend` on `get_current_user`). The SDK default is the
+  API-key header — same credentials as v1, no token lifecycle. Bearer JWT via
+  `POST /api/v2/token` (username/password, 30-min access / 30-day refresh) is also
+  supported; the transport transparently refreshes once on 401, then re-raises.
+- **Sync facade.** `Client` mirrors every `AsyncClient` method by dispatching to a
+  background-thread event loop (a "portal"), not `asyncio.run` — so sync usage works
+  inside Jupyter, where a loop is already running. Sync mirrors are generated
+  mechanically from the async resource surface, not hand-maintained.
+- **Bulk pipeline.** `records.bulk_upsert()` auto-chunks at the server's 500-item
+  cap and dispatches chunks with bounded concurrency (default semaphore of 4,
+  configurable) plus an optional progress callback. Deletes chunk at the 100-id cap.
+  Pushing 50k records is one SDK call.
+- **Immutable-data caching.** Schema versions are immutable → cached in-client by
+  `(schema_id, version)`. Question name↔id maps cached per schema, invalidated on
+  question writes through the same client.
+
+## 4. Domain resources — the extraction loop
+
+The resource layer is the anti-corruption layer: every wire quirk lives here once,
+so nothing above it knows about `snake_case` bodies, double-wrapping, or id-vs-name
+keying.
+
+```python
+async with AsyncClient(api_url=..., api_key=...) as xl:
+    schema  = await xl.schemas.get_by_name(workspace_id, "clinical_trials")
+    version = await xl.schemas.publish(schema.id, pandera_schema, review_widgets={...})
+    await xl.records.bulk_upsert(schema.id, rows, reference="10.1000/xyz")   # auto-chunked
+    page    = await xl.records.search(schema.id, text="tumor size",
+                                      filters=[("patient_age", "ge", 18)])
+    await xl.suggestions.upsert(record.id, question="dosage", value=..., score=0.9,
+                                agent="claude")
+    proj    = await xl.projections.get(workspace_id, reference="10.1000/xyz")
+```
+
+Resources and the quirks they encapsulate (all confirmed against the server and the
+frontend's PR #230 findings):
+
+- **Schemas** — create/get/list/update; `publish()` posts a Pandera body plus
+  out-of-band `review_widgets` (Pandera's `to_json()` drops column metadata);
+  versions list/get; `columns()` from the current version's cache.
+- **Records** — `bulk_upsert` (idempotent on `external_id`, `metadata` is
+  patch-like), `search` (text + `(column, op, value)` filters with
+  `op ∈ {eq,in,ge,le}`; `total` is approximate — stale index ids are skipped and
+  FTS saturates, so pagination never promises exact counts), `list`, `delete`,
+  and `references.get()` for the cross-schema reference view. `bulk_upsert`
+  accepts list-of-dicts or a pandas DataFrame (pandas imported lazily inside that
+  path only).
+- **Questions** — list/get per schema. Callers address questions by **name**;
+  the resource resolves to **id** via the cached map (suggestions key by id;
+  cells and response values key by name — getting this wrong silently detaches
+  provenance).
+- **Suggestions** — upsert per (record, question) with `value`, `score`, `agent`,
+  `type`; list per record.
+- **Projections** — `get(workspace_id, reference)` → cells of response ∨ suggestion
+  per question with `source` provenance. **Responses (read-only this slice)** —
+  `GET` returns literal `null` with 200 when absent (mapped to `None`, not an
+  error); values are double-wrapped `{name: {"value": …}}` and unwrapped here.
+
+Domain classes are thin Pydantic models with behavior (e.g. `SchemaVersion.find_column`,
+`SearchPage.records`), constructed from DTOs by the resources — same layering the
+frontend proved testable without a live backend.
+
+## 5. Agentic CLI
+
+v2 commands register at the **top level** of the existing `extralit` app — no `v2`
+prefix in the interface — built over the sync facade:
+
+```
+extralit schemas     list | get | create | publish | versions
+extralit records     upsert | search | list | delete
+extralit questions   list
+extralit suggestions upsert
+extralit projection  get
+extralit references  get
+```
+
+The v2 model owns the CLI surface going forward. Consequence: the existing v1
+`extralit schemas` subcommand (workspace-file Pandera management) is **replaced** by
+the v2 implementation — a deliberate breaking change; the CLI leads the v1→v2
+transition even while the SDK library keeps the v1 surface intact. All other v1
+subcommands (`datasets`, `users`, `workspaces`, `extraction`, …) remain untouched
+until Phase 6.
+
+- **JSON-first.** Every command accepts `--json`; when stdout is not a TTY, JSON is
+  the default automatically. JSON shapes are the serialized DTOs — stable because
+  they are drift-gated by the contract layer. Rich tables remain the TTY default
+  for humans.
+- **Exit-code and stream contract.** stdout carries data only; errors are structured
+  JSON on stderr (`{"error": {"type", "status", "detail"}}`). Exit codes:
+  0 success, 1 API/runtime error, 2 usage error, 3 validation (422).
+- **Non-interactive by construction.** Env vars (`EXTRALIT_API_URL`,
+  `EXTRALIT_API_KEY` or username/password) and the credentials file; no prompt is
+  ever emitted when stdin is not a TTY.
+- **Piping.** `records upsert` reads JSONL from stdin or `--file`; `records search
+  --limit/--offset` page through cleanly for chaining.
+- **Startup budget.** The v2 commands register into `cli/app.py` via lazy callbacks;
+  heavy deps (pandas, pandera, datasets) import only inside the commands that use
+  them. Budget: `extralit schemas --help` completes in < 300 ms on the Orin dev
+  host, verified with `python -X importtime` in a CI check.
+
+## 6. Error handling
+
+`V2APIError(status, detail)` base → `AuthError` (post-refresh 401),
+`NotFoundError`, `ValidationError` (normalizes both server 422 body shapes —
+`detail: str` and `detail: [{loc, msg}]`). The CLI maps this hierarchy onto the
+exit-code contract. Search totals are documented as approximate rather than
+pretending exactness.
+
+## 7. Testing
+
+- **Unit (no server):** pytest + pytest-asyncio + pytest-httpx. Resource tests pin
+  each wire quirk by name: double-wrap/unwrap, name↔id suggestion join,
+  both 422 shapes, null-response-200, bulk chunking + bounded concurrency +
+  mid-batch failure behavior, token refresh on 401. Sync-facade smoke tests run
+  inside a live event loop (Jupyter simulation). CLI tests via Typer's runner
+  assert `--json` output shapes, stderr error JSON, and exit codes.
+- **CI gates:** codegen no-drift, snapshot-vs-server, import-boundary grep
+  (`extralit/v2` must not import v1 modules), CLI startup-time budget.
+- **Integration (opt-in marker, live stack):** one end-to-end extraction-loop test
+  (publish → upsert → suggest → search → projection) reusing the deterministic
+  seeding approach from the frontend's `e2e/v2/`.
+
+## Scope
+
+**In this slice:** everything above — contract layer, transport + sync facade,
+the five resources, the top-level CLI verbs with JSON-first output (including the
+replacement of the v1 `schemas` subcommand), unit tests + CI gates.
+
+**Deferred by design:** responses *write* (submit/draft/discard — the review loop
+belongs to a follow-up once the frontend's Phase 5 queue lands), DataFrame/parquet/HF
+export, `rebuild-index` and other admin verbs, webhooks, span questions, and the
+retirement of the remaining v1 CLI subcommands and SDK surface (Phase 6) this
+boundary exists to enable.

extralit-frontend/e2e/v2/draft-lifecycle.spec.ts

@@ -6,7 +6,7 @@ import { createIsolatedRecord, expect, loadSeed, signIn, test } from "./fixtures
 // submit at the end never contaminates another spec's clean-suggestion precondition.
 test("draft persists in the form without touching the projection, then submits", async ({ page, request }) => {
   const seed = loadSeed();
-  const { reference, recordId } = await createIsolatedRecord(request, `${seed.reference}-draft-lifecycle`);
+  const { reference, recordId } = await createIsolatedRecord(request, "10.2000/e2e-draft-lifecycle");
 
   await signIn(page);
   await page.goto(`/references/${encodeURIComponent(reference)}?workspace_id=${seed.workspaceId}`);

extralit-frontend/e2e/v2/fixtures.ts

@@ -32,6 +32,9 @@ export const apiToken = async (request: APIRequestContext): Promise<string> => {
 // each seed one of these in beforeEach so they never share the single seed record — which
 // otherwise races in parallel and, in serial order, leaves a submitted response that breaks
 // the next spec's clean "Suggestion" precondition (roborev job 154). A reseed wipes the schema.
+// The reference must NOT contain seed.reference as a substring: other specs assert on the
+// shared record via non-exact getByText(seed.reference), and a superstring reference in the
+// same schema substring-matches those assertions into strict-mode violations (roborev job 157).
 export const createIsolatedRecord = async (
   request: APIRequestContext,
   reference: string

extralit-frontend/e2e/v2/review-loop.spec.ts

@@ -5,7 +5,7 @@ import { createIsolatedRecord, expect, loadSeed, signIn, test } from "./fixtures
 // Uses its own isolated record so it never shares mutable response state with other specs.
 test("converts a suggestion into a submitted response", async ({ page, request }) => {
   const seed = loadSeed();
-  const { reference, recordId } = await createIsolatedRecord(request, `${seed.reference}-review-loop`);
+  const { reference, recordId } = await createIsolatedRecord(request, "10.2000/e2e-review-loop");
 
   await signIn(page);
   await page.goto(`/references/${encodeURIComponent(reference)}?workspace_id=${seed.workspaceId}`);

extralit/CLAUDE.md

@@ -47,3 +47,14 @@ src/extralit/
   /models      # Data models
   /utils       # Utilities
 ```
+
+## v2 SDK (`src/extralit/v2/`)
+
+- Parallel package for `/api/v2` (schema-centric). Import wall: v2 imports nothing from v1
+  except `extralit.client.login`; only `cli/app.py` imports v2 (composition root).
+- Wire types are GENERATED: `_api/openapi.json` (server `openapi-dump` snapshot) ->
+  `_api/_generated.py` via datamodel-codegen. Never hand-edit; regenerate with the command
+  in `tests/unit/v2/test_contract.py` and keep both in sync (drift-gated).
+- `AsyncClient` is the real client; `Client` is a mechanical sync facade (background-thread
+  portal — works in Jupyter). CLI verbs register at TOP level (`extralit schemas|records|...`),
+  JSON-first (`--json` or non-TTY), errors as JSON on stderr (exit 0/1/2/3).

extralit/pyproject.toml

@@ -53,6 +53,7 @@ root = "./src"
 
 [tool.ruff]
 line-length = 120
+extend-exclude = ["src/extralit/v2/_api/_generated.py"]
 
 [tool.ruff.lint]
 select = [
@@ -115,6 +116,7 @@ dev = [
     "mknotebooks >= 0.8.0",
     "pytest-retry>=1.5",
     "ty>=0.0.46",
+    "datamodel-code-generator>=0.26,<0.69",
 ]
 
 [project.scripts]

extralit/src/extralit/cli/app.py

@@ -9,7 +9,6 @@
     info,
     login,
     logout,
-    schemas,
     training,
     users,
     whoami,
@@ -50,13 +49,17 @@ def register_subcommands():
     app.add_typer(info.app, name="info")
     app.add_typer(login.app, name="login")
     app.add_typer(logout.app, name="logout")
-    app.add_typer(schemas.app, name="schemas")
     app.add_typer(training.app, name="training")
     app.add_typer(users.app, name="users")
     app.add_typer(whoami.app, name="whoami")
     app.add_typer(workflows.app, name="workflows")
     app.add_typer(workspaces.app, name="workspaces")
 
+    # v2 owns the top-level verbs: schemas, records, questions, suggestions, projection, references.
+    from extralit.v2.cli import add_v2_commands  # composition-root exception to the v1/v2 wall
+
+    add_v2_commands(app)
+
 
 register_subcommands()
 

extralit/src/extralit/v2/__init__.py

@@ -0,0 +1,33 @@
+from extralit.v2._sync import Client
+from extralit.v2.client import AsyncClient
+from extralit.v2.models import (
+    ProjectionCell,
+    ProjectionRecord,
+    ProjectionView,
+    Question,
+    Record,
+    ReferenceGroup,
+    ReferenceView,
+    Response,
+    Schema,
+    SchemaVersion,
+    SearchPage,
+    Suggestion,
+)
+
+__all__ = [
+    "AsyncClient",
+    "Client",
+    "ProjectionCell",
+    "ProjectionRecord",
+    "ProjectionView",
+    "Question",
+    "Record",
+    "ReferenceGroup",
+    "ReferenceView",
+    "Response",
+    "Schema",
+    "SchemaVersion",
+    "SearchPage",
+    "Suggestion",
+]

extralit/src/extralit/v2/_api/_errors.py

@@ -0,0 +1,55 @@
+from __future__ import annotations
+
+from typing import Any
+
+
+class V2APIError(Exception):
+    """Base error for /api/v2 calls."""
+
+    def __init__(self, status_code: int, detail: Any = None):
+        self.status_code = status_code
+        self.detail = detail
+        super().__init__(f"HTTP {status_code}: {detail!r}")
+
+
+class AuthError(V2APIError):
+    """401/403, raised after any transparent token refresh has already been attempted."""
+
+
+class NotFoundError(V2APIError):
+    """404."""
+
+
+class ValidationError(V2APIError):
+    """422. The server emits two body shapes (detail: str | list[{loc, msg}]); `.errors` is normalized."""
+
+    def __init__(self, status_code: int, detail: Any = None):
+        super().__init__(status_code, detail)
+        self.errors = normalize_validation_detail(detail)
+
+
+def normalize_validation_detail(detail: Any) -> list[dict]:
+    if detail is None:
+        return []
+    if isinstance(detail, str):
+        return [{"loc": [], "msg": detail}]
+    if isinstance(detail, list):
+        out = []
+        for item in detail:
+            if isinstance(item, dict):
+                out.append({"loc": list(item.get("loc", [])), "msg": str(item.get("msg", item))})
+            else:
+                out.append({"loc": [], "msg": str(item)})
+        return out
+    return [{"loc": [], "msg": str(detail)}]
+
+
+def error_from_response(status_code: int, body: Any) -> V2APIError:
+    detail = body.get("detail") if isinstance(body, dict) else body
+    if status_code in (401, 403):
+        return AuthError(status_code, detail)
+    if status_code == 404:
+        return NotFoundError(status_code, detail)
+    if status_code == 422:
+        return ValidationError(status_code, detail)
+    return V2APIError(status_code, detail)