Research-grade eval: live model benchmark, judge calibration, structured-output study (v2.2.0)

.gitignore

@@ -13,9 +13,9 @@ web-ext.config.ts
 !.env.example
 *.local
 
-# Eval harness artifacts (generated reports/caches)
+# Eval harness artifacts (generated checkpoints/pages; reports (.md) are committed)
 eval/.cache/
-eval/results/*.json
+eval/results/*
 !eval/results/.gitkeep
 
 # Editor / OS

CHANGELOG.md

@@ -5,6 +5,44 @@ All notable changes to this project are documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [2.2.0] - 2026-07-10
+
+Research-grade evaluation: live model benchmark, judge calibration, and a
+structured-output study — which caught and fixed a product-breaking
+reasoning-model bug.
+
+### Added
+
+- Live translation benchmark (`pnpm bench`): 4 local models × 2 prompt
+  conditions × 27 curated EN→zh-TW fixtures with Taiwan-convention references;
+  reference-based chrF (sacrebleu-cross-validated), artifact rates,
+  TTFT-net/TTFT-UI latency, tokens/s, and a JSON-schema-constrained LLM judge
+  → `eval/BENCHMARK-RESULTS.md`
+- Judge calibration workflow (`pnpm bench:agreement`): seeded blind labeling
+  page + Cohen's κ (plain & quadratic-weighted) judge↔human → `eval/AGREEMENT.md`
+- Structured-output study (`pnpm eval:structured`): prompt-only vs
+  schema-constrained decoding across 4 models × 16 realistic capture excerpts,
+  with a failure-shape taxonomy → `eval/STRUCTURED-RESULTS.md`
+- `docs/BENCHMARK.md` — methodology, found-bug case study, limitations
+
+### Changed
+
+- **Ollama client migrated from OpenAI-compat `/v1/chat/completions` to native
+  `/api/chat` with `think: false`** (now requires Ollama ≥ 0.9)
+- Default model `qwen2.5` → `qwen3:latest` — chosen by the benchmark (best
+  chrF at the lowest TTFT of the models measured)
+- Capture enrichment now uses schema-constrained decoding (Ollama `format`) —
+  the study measured it closing the last unreliable tail at zero latency
+  cost; the tolerant parser remains as content hygiene
+
+### Fixed
+
+- Reasoning models (qwen3 family, deepseek-r1) produced **no visible output**
+  through the compat endpoint — chain-of-thought consumed the entire
+  generation in a separate `reasoning` field (measured: 99 s / 4,055 tokens /
+  0 visible characters on one fixture; 1.6 s after the fix). Found by the
+  benchmark harness.
+
 ## [2.1.0] - 2026-07-09
 
 Switch to local Ollama backend.

CONTRIBUTING.md

@@ -48,10 +48,14 @@ The translation reliability harness runs offline against a fixed dataset:
 pnpm eval
 ```
 
-The optional live LLM-as-judge (`eval/judge.ts`) calls a real model instead of
-the fixed dataset. It needs a local [Ollama](https://ollama.com/) server —
-install Ollama, then `ollama pull qwen2.5` — and reads the server URL from the
-`OLLAMA_URL` env var (default `http://localhost:11434`); no API key required.
+The live evaluation suite — `pnpm bench` (model × prompt benchmark),
+`pnpm eval:structured` (structured-output study), `pnpm bench:agreement`
+(judge↔human calibration), and the older `eval/judge.ts` — calls real models
+instead of fixed datasets. It needs a local [Ollama](https://ollama.com/)
+server (≥ 0.9) — install Ollama, then `ollama pull qwen3` — and reads the
+server URL from the `OLLAMA_URL` env var (default `http://localhost:11434`);
+no API key required. See [`docs/BENCHMARK.md`](docs/BENCHMARK.md) for the
+methodology.
 
 ## Project layout
 

README.md

@@ -68,7 +68,40 @@ _Measured over 23 curated fixtures (21 Traditional-Chinese targets). Regenerate
 with `pnpm eval`; full report in [`eval/RESULTS.md`](eval/RESULTS.md)._
 
 The pure core carries **100% function coverage and ~94% line coverage** across
-85 unit tests (`pnpm test:cov`).
+120 unit tests (`pnpm test:cov`).
+
+## Which local model? — live benchmark
+
+The offline eval freezes model output to score the pipeline; `pnpm bench`
+asks the opposite question against live models: **which model should you run,
+and what does each design choice cost?** 27 curated EN→zh-TW fixtures with
+Taiwan-convention references × 4 models × 2 prompt conditions, streamed
+through the exact shipped pipeline and scored with sacrebleu-cross-validated
+chrF, artifact detectors, latency probes, and a schema-constrained LLM judge
+(itself calibrated against human labels — Cohen's κ workflow included).
+
+| Model | chrF ↑ | TTFT-UI p50 | Tokens/s | Verdict |
+| --- | --- | --- | --- | --- |
+| **qwen3** (default) | **46.3** | **451 ms** | 48 | best quality/latency balance |
+| qwen3.5 | 43.4 | 730 ms | 42 | no chrF edge, 1.6× the wait |
+| llama3.1 | 31.6 | 532 ms | 49 | fast, but ~13 chrF behind |
+| deepseek-r1:8b | 36.6 | 6,353 ms | — | 6-second "thinking tax" — wrong workload |
+
+_Engineered-prompt condition, seed 42; full tables in
+[`eval/BENCHMARK-RESULTS.md`](eval/BENCHMARK-RESULTS.md), methodology and
+limitations in [`docs/BENCHMARK.md`](docs/BENCHMARK.md)._
+
+Two findings worth calling out:
+
+- **The benchmark caught a product-breaking bug.** Through Ollama's
+  OpenAI-compat endpoint, reasoning models can spend the *entire* generation
+  on hidden chain-of-thought — one fixture: 99 s, 4,055 tokens, zero visible
+  characters. The client now uses the native `/api/chat` with `think: false`
+  (same fixture: 1.6 s).
+- **The reliability layer is a measured tradeoff, not free.** It zeroes
+  preamble on dirty outputs and halves deepseek-r1's Simplified leakage, but
+  costs ~200 ms of first paint and a fraction of a chrF point on clean
+  outputs — numbers, not vibes, either way.
 
 ## How it works
 
@@ -87,8 +120,13 @@ The pure core carries **100% function coverage and ~94% line coverage** across
 - **Cancellation-safe streaming**
   ([`src/api/ollama.ts`](src/api/ollama.ts) +
   [`src/entrypoints/background.ts`](src/entrypoints/background.ts)) — each
-  request owns an `AbortController`; a new selection or a closed panel aborts the
-  in-flight SSE with no shared mutable state to race on.
+  request owns an `AbortController`; a new selection or a closed panel aborts
+  the in-flight stream with no shared mutable state to race on.
+- **Reasoning-model safe** — the client uses Ollama's native `/api/chat` with
+  `think: false` because the benchmark caught the OpenAI-compat endpoint
+  burning entire generations as hidden reasoning with zero visible output on
+  qwen3-family and deepseek-r1 models
+  ([`docs/BENCHMARK.md`](docs/BENCHMARK.md) §6). Requires Ollama ≥ 0.9.
 - **Fully local** — no cloud, no key, no telemetry. The selected text is sent
   only to a local Ollama server on your machine; nothing leaves your device.
   The server URL lives in `chrome.storage` and is read only by the background
@@ -121,20 +159,20 @@ cheap, reliable part on-device and defers the expensive part — rather than
 re-implementing a knowledge base it has no business owning.
 
 Optionally, a small local model can pre-label a capture with a title, summary,
-and tags. Small models are unreliable at structured output, so this is strictly
-best-effort — and _measured_: `parseEnrichResponse` salvages usable metadata
-from fenced, preamble-wrapped, and trailing-prose replies that a naive
-`JSON.parse` drops.
+and tags. That pipeline is defense-in-depth, and every layer is _measured_:
 
-| Metric                                              | Rate       |
-| --------------------------------------------------- | ---------- |
-| Naive `JSON.parse` yields an object                 | 42.9%      |
-| Robust `parseEnrichResponse` yields usable metadata | **71.4%**  |
+| Layer | Evidence |
+| --- | --- |
+| Schema-constrained decoding (Ollama `format`) | took the one imperfect model (deepseek-r1) from 93.3% → **100%** usable metadata at zero latency cost — live study over 4 models × 16 excerpts ([`eval/STRUCTURED-RESULTS.md`](eval/STRUCTURED-RESULTS.md)) |
+| Tolerant `parseEnrichResponse` | salvages 71.4% vs naive parsing's 42.9% on an archive of 14 hostile reply shapes from older/thinking models ([`eval/CAPTURE-RESULTS.md`](eval/CAPTURE-RESULTS.md)); today it mostly does content hygiene — length caps, tag normalisation |
+| `status: raw` handoff | even a perfect-looking label is garnish; the raw capture stays the source of truth |
 
-_Offline, deterministic run over 14 real small-model reply shapes; regenerate
-with `pnpm eval:capture`, full report in
-[`eval/CAPTURE-RESULTS.md`](eval/CAPTURE-RESULTS.md). This is why enrichment is
-off by default — the raw capture is always the source of truth._
+The live study also produced an honest negative result: with the shipped
+prompt, temperature 0, and thinking disabled, **modern small models emit
+clean JSON ~100% of the time** — the dramatic salvage rates belong to older
+model generations. Enrichment stays off by default anyway: it adds a
+model round-trip per capture, and reasoning-class models take ~45 s to label
+a paragraph (measured), which no capture UX survives.
 
 Set your vault, capture folder, and the enrichment toggle in the popup; leave
 the vault blank to use whichever vault is currently open.
@@ -155,7 +193,8 @@ OpenRead translates through a local [Ollama](https://ollama.com/) server — no
 API key required.
 
 1. [Install Ollama](https://ollama.com/).
-2. Pull a model: `ollama pull qwen2.5`.
+2. Pull a model: `ollama pull qwen3` (the benchmarked default; see
+   [`docs/BENCHMARK.md`](docs/BENCHMARK.md) for how it was chosen).
 3. Start the server: `ollama serve`.
 4. Allow the extension's origin through Ollama's CORS by setting
    `OLLAMA_ORIGINS=chrome-extension://*` before starting it:
@@ -164,7 +203,7 @@ API key required.
    - **Windows**: set a user environment variable `OLLAMA_ORIGINS=chrome-extension://*`, then restart Ollama
 
 Open the toolbar popup and set the Ollama server URL (default
-`http://localhost:11434`) and model (default `qwen2.5`), plus a target
+`http://localhost:11434`) and model (default `qwen3:latest`), plus a target
 language.
 
 ## Usage

docs/ARCHITECTURE.md

@@ -34,9 +34,19 @@ public/
 eval/
   dataset/fixtures.json         curated translation failure-mode fixtures
   dataset/capture-fixtures.json curated small-model enrichment reply shapes
+  dataset/bench-fixtures.json   27 EN→zh-TW segments with Taiwan-convention references
+  dataset/enrich-inputs.json    16 realistic capture excerpts (EN/zh-TW/mixed)
   detectors.ts                  preamble / Simplified / echo detectors (reuse core)
-  run.ts                        translation before→after runner → eval/RESULTS.md
-  capture-run.ts                enrichment-parser runner → eval/CAPTURE-RESULTS.md
+  run.ts                        offline before→after runner → eval/RESULTS.md (CI gate)
+  capture-run.ts                offline enrichment-parser runner → eval/CAPTURE-RESULTS.md
+  bench/                        live model × prompt benchmark (needs Ollama, not CI)
+    chrf.ts                       chrF metric, sacrebleu-cross-validated
+    kappa.ts                      Cohen's κ (plain + quadratic-weighted)
+    run.ts                        matrix runner + LLM judge → eval/BENCHMARK-RESULTS.md
+    agreement.ts                  human-labeling page + judge↔human κ → eval/AGREEMENT.md
+  structured/                   live structured-output study (needs Ollama, not CI)
+    taxonomy.ts                   failure-shape classifier for small-model replies
+    run.ts                        prompt vs schema-constrained → eval/STRUCTURED-RESULTS.md
 ```
 
 ### Why this split
@@ -56,14 +66,22 @@ eval/
 A selection triggers a long-lived port to the background worker, which owns the
 network call and streams cleaned chunks back. Each request has its own
 `AbortController`, so a new selection (or a closed panel) cancels the previous
-SSE cleanly.
+stream cleanly.
+
+The client talks to Ollama's **native `/api/chat`** endpoint (NDJSON streaming)
+with `think: false`. The benchmark found that the OpenAI-compat `/v1` endpoint
+routes reasoning-model chain-of-thought into a separate field and can leave
+`content` empty for the entire generation (qwen3.5: 99 s, 4,055 tokens, zero
+visible output) — the native endpoint disables thinking on hybrid models and
+keeps it out of `content` on models that cannot stop (deepseek-r1). Requires
+Ollama ≥ 0.9. See [`BENCHMARK.md`](BENCHMARK.md) §6.
 
 ```mermaid
 sequenceDiagram
     participant U as User
     participant C as Content script (ui/selection.ts)
     participant B as Background broker
-    participant O as Ollama (local, SSE)
+    participant O as Ollama (local, /api/chat NDJSON)
 
     U->>C: select text, click 文
     C->>C: shouldBypassAI(text, target)?
@@ -72,14 +90,14 @@ sequenceDiagram
     else needs translation
         C->>B: connect "stream-translate" + START_STREAM {text, target, model}
         B->>B: load Ollama base URL from storage; new AbortController
-        B->>O: POST /chat/completions (stream:true, signal)
-        loop each SSE delta
-            O-->>B: data: {delta.content}
+        B->>O: POST /api/chat (stream:true, think:false, signal)
+        loop each NDJSON chunk
+            O-->>B: {message:{content}, done:false}
             B->>B: StreamAssembler.push (buffer opening,<br/>strip preamble, SC→TC)
             B-->>C: {status:"streaming", chunk}
             C-->>U: append chunk to panel
         end
-        O-->>B: data: [DONE]
+        O-->>B: {done:true, eval_count}
         B-->>C: {status:"done"}
     end
 
@@ -95,5 +113,7 @@ sequenceDiagram
 | OpenCC `s2twp` over a hand-rolled map                                           | v1's unconditional character map corrupted common words (`界面→界麵`). Phrase-level conversion is correct and maintained. |
 | Ollama base URL read in background from storage; request goes to a local server | No secret rides the message bus — and since the request never leaves the machine, there's nothing to leak either way.     |
 | Offline, deterministic eval                                                     | Before/after numbers are reproducible in CI without a running Ollama server, so they are honest to cite.                  |
+| Native `/api/chat` + `think: false`, not the OpenAI-compat `/v1`                | On reasoning models `/v1` can burn the whole generation as hidden `reasoning` with `content` empty — the user sees nothing. Found by the benchmark; see `docs/BENCHMARK.md` §6. |
+| Live benchmark imports the shipped modules                                      | `pnpm bench` scores `buildMessages`/`extractChunk`/`StreamAssembler` themselves, so its numbers describe the product, not a re-implementation. |
 | Capture writes via `obsidian://new` from the content script, not a new API      | Keeps least-privilege intact (no `downloads`/native-host permission) and reuses the user gesture; oversized notes fall back to the clipboard. |
 | Captures are `status: raw`; heavy synthesis deferred to a downstream model      | On-device small models fail at structured output (measured), so OpenRead ships a reliable raw note and lets a stronger "second brain" process it. |