Research-grade eval: live model benchmark, judge calibration, structured-output study (v2.2.0)

[0Smallcat0]

What

Turns the eval story from "offline sanitizer scoring" into a research-grade evaluation suite — and ships the product fixes the data demanded.

1. Live model × prompt benchmark (pnpm bench)

4 local models × 2 prompt conditions × 27 curated EN→zh-TW fixtures (Taiwan-convention references: 升息一碼, 執行緒, 連接埠, 工作階段…), streamed through the exact shipped pipeline (buildMessages → /api/chat NDJSON → extractChunk → StreamAssembler + OpenCC). Scored with:

• chrF — implemented in-repo, cross-validated against Python sacrebleu 2.6.0 to 6 decimals
• artifact detectors (preamble / echo / Simplified leak), raw vs piped
• TTFT-net vs TTFT-UI — prices the reluctant buffer (~200 ms)
• LLM judge (JSON-schema-constrained, temp 0) + human-calibration workflow: seeded blind labeling page + Cohen's κ (plain & quadratic) via pnpm bench:agreement

Result highlights (216/216 cells, 0 errors → eval/BENCHMARK-RESULTS.md):

Model	chrF ↑	TTFT-UI p50	Verdict
qwen3	46.3	451 ms	new default
qwen3.5	43.4	730 ms	no edge, 1.6× wait
llama3.1	31.6	532 ms	~13 chrF behind
deepseek-r1:8b	36.6	6,353 ms	thinking tax

2. Found & fixed: reasoning models returned nothing

Through the OpenAI-compat /v1 endpoint, thinking models can burn the whole generation as hidden reasoning with content empty — measured: qwen3.5, 99 s, 4,055 tokens, 0 visible characters. think:false via /v1 didn't help.

Fix: client migrated to native /api/chat + think: false (same fixture: 1.6 s / TTFT 282 ms). Hybrid thinkers stop thinking; non-thinkers no-op; deepseek-r1 still thinks but keeps it out of content. Requires Ollama ≥ 0.9.

3. Structured-output study (pnpm eval:structured)

Prompt-only vs schema-constrained decoding, 4 models × 16 realistic excerpts + failure-shape taxonomy. Honest surprise: modern small models emit clean JSON ~100% of the time with the shipped prompt at temp 0 — the old 42.9%→71.4% salvage story belongs to older/thinking-contaminated generations. Schema still closes the last tail (deepseek-r1 93.3%→100%) at zero latency cost → now sent on every enrichment request.

4. Data-driven product changes

• default model qwen2.5 → qwen3:latest
• enrichment schema-constrained
• docs/BENCHMARK.md — methodology, found-bug case study, limitations, reproduce steps

Verification

• pnpm compile / pnpm lint clean; pnpm test 120 passed (+35)
• pnpm eval, pnpm eval:capture unchanged (offline CI gates stay deterministic)
• pnpm build production build OK
• Full live runs on RTX 4060 Laptop: bench 216/216 cells 0 errors, structured 128 cells 1 transient timeout (deepseek-r1, retried clean)

Human labeling (15 min, eval/results/labeling.html → pnpm bench:agreement) is the one follow-up that needs a person; infrastructure and docs are in place.

🤖 Generated with Claude Code