Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,11 @@ jobs:

- uses: pnpm/action-setup@v4
with:
version: 9
version: 11

- uses: actions/setup-node@v4
with:
node-version: 20
node-version: 22
cache: 'pnpm'

- run: pnpm install --frozen-lockfile
Expand Down
39 changes: 37 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,8 +65,8 @@ Ollama server, no network — so the numbers are reproducible in CI.
_Measured over 23 curated fixtures (21 Traditional-Chinese targets). Regenerate
with `pnpm eval`; full report in [`eval/RESULTS.md`](eval/RESULTS.md)._

The pure core carries **100% function coverage and ~92% line coverage** across
57 unit tests (`pnpm test:cov`).
The pure core carries **100% function coverage and ~94% line coverage** across
85 unit tests (`pnpm test:cov`).

## How it works

Expand Down Expand Up @@ -95,6 +95,40 @@ The pure core carries **100% function coverage and ~92% line coverage** across
See [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) for the module map and the
streaming sequence diagram.

## Capture to Obsidian

Reading is only half the loop — OpenRead also turns any translated selection
into a note in your [Obsidian](https://obsidian.md) vault. Once a translation
streams in, a **+ 儲存到 Obsidian** button drops a Markdown note — original,
translation, and a machine-readable YAML header — straight into your vault via
an `obsidian://new` URI. No extra permissions, no server; notes too large for a
protocol-handler URL fall back to the clipboard.

That header is a deliberate **handoff contract**. Every note is written
`status: raw`, so a stronger downstream model (a "second brain") can query the
unprocessed captures, synthesize them, and flip the flag. OpenRead does the
cheap, reliable part on-device and defers the expensive part — rather than
re-implementing a knowledge base it has no business owning.

Optionally, a small local model can pre-label a capture with a title, summary,
and tags. Small models are unreliable at structured output, so this is strictly
best-effort — and _measured_: `parseEnrichResponse` salvages usable metadata
from fenced, preamble-wrapped, and trailing-prose replies that a naive
`JSON.parse` drops.

| Metric | Rate |
| --------------------------------------------------- | ---------- |
| Naive `JSON.parse` yields an object | 42.9% |
| Robust `parseEnrichResponse` yields usable metadata | **71.4%** |

_Offline, deterministic run over 14 real small-model reply shapes; regenerate
with `pnpm eval:capture`, full report in
[`eval/CAPTURE-RESULTS.md`](eval/CAPTURE-RESULTS.md). This is why enrichment is
off by default — the raw capture is always the source of truth._

Set your vault, capture folder, and the enrichment toggle in the popup; leave
the vault blank to use whichever vault is currently open.

## Install (from source)

```bash
Expand Down Expand Up @@ -137,6 +171,7 @@ pnpm dev # HMR dev build (Chrome); pnpm dev:firefox for Firefox
pnpm test # Vitest unit suite
pnpm test:cov # …with coverage
pnpm eval # reliability eval -> eval/RESULTS.md
pnpm eval:capture # capture-enrichment eval -> eval/CAPTURE-RESULTS.md
pnpm compile # tsc --noEmit (strict)
pnpm lint # ESLint
pnpm build # production build -> .output/chrome-mv3
Expand Down
17 changes: 12 additions & 5 deletions docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,24 +14,29 @@ src/
stream.ts StreamAssembler — reluctant-buffer streaming logic
zh-convert.ts OpenCC s2twp Simplified→Traditional wrapper
prompt.ts system prompt + adaptive anti-echo few-shot
capture.ts capture → Markdown + obsidian:// URI builders
enrich.ts enrich prompt + tolerant small-model JSON salvager
types.ts shared domain types
api/
ollama.ts typed streaming + single-shot client, pure extractDelta
ollama.ts typed streaming + single-shot + enrich client
messaging.ts typed content⇄background port + one-shot protocol
settings.ts typed chrome.storage.sync wrapper
ui/
selection.ts shared selection icon + panel + stream client
capture.ts capture orchestrator: enrich round-trip + write
entrypoints/
background.ts service worker: stream broker + PDF router
background.ts service worker: stream broker + PDF router + enrich
content.ts mounts the selection translator on web pages
pdf-viewer.ts mounts the same translator inside the PDF.js viewer
popup/ settings UI (index.html + main.ts)
public/
pdfjs/ vendored PDF.js viewer (worker path fixed to .mjs)
eval/
dataset/fixtures.json curated failure-mode fixtures
detectors.ts preamble / Simplified / echo detectors (reuse core)
run.ts before→after runner → eval/RESULTS.md
dataset/fixtures.json curated translation failure-mode fixtures
dataset/capture-fixtures.json curated small-model enrichment reply shapes
detectors.ts preamble / Simplified / echo detectors (reuse core)
run.ts translation before→after runner → eval/RESULTS.md
capture-run.ts enrichment-parser runner → eval/CAPTURE-RESULTS.md
```

### Why this split
Expand Down Expand Up @@ -90,3 +95,5 @@ sequenceDiagram
| OpenCC `s2twp` over a hand-rolled map | v1's unconditional character map corrupted common words (`界面→界麵`). Phrase-level conversion is correct and maintained. |
| Ollama base URL read in background from storage; request goes to a local server | No secret rides the message bus — and since the request never leaves the machine, there's nothing to leak either way. |
| Offline, deterministic eval | Before/after numbers are reproducible in CI without a running Ollama server, so they are honest to cite. |
| Capture writes via `obsidian://new` from the content script, not a new API | Keeps least-privilege intact (no `downloads`/native-host permission) and reuses the user gesture; oversized notes fall back to the clipboard. |
| Captures are `status: raw`; heavy synthesis deferred to a downstream model | On-device small models fail at structured output (measured), so OpenRead ships a reliable raw note and lets a stronger "second brain" process it. |
14 changes: 14 additions & 0 deletions eval/CAPTURE-RESULTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# OpenRead — Capture Enrichment Eval

Offline, deterministic run over **14** small-model reply fixtures. No network, no Ollama server needed.

| Metric | Count | Rate |
| --- | --- | --- |
| Naive `JSON.parse` yields an object | 6 | 42.9% |
| Robust `parseEnrichResponse` yields usable metadata | 10 | 71.4% |
| Usable title recovered | 9 | 64.3% |
| Usable summary recovered | 9 | 64.3% |
| ≥1 tag recovered | 9 | 64.3% |

_The robust parser recovers usable metadata from **5** replies that a naive `JSON.parse` drops (fenced, preamble-wrapped, or trailing-prose output), while rejecting empty/garbage replies that naive parsing would wave through. Enrichment is best-effort — every capture writes a raw note regardless._

93 changes: 93 additions & 0 deletions eval/capture-run.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
/**
* Capture-enrichment eval — deterministic and offline.
*
* Small local models return metadata unreliably: wrapped in code fences, buried
* in preamble, tags as a comma string, or nothing usable at all. This measures
* how much the pure `parseEnrichResponse` salvager improves on a naive
* `JSON.parse` over a fixture set of real small-model reply shapes. No network,
* no Ollama server — reproducible in CI and honest to cite. Run with
* `pnpm eval:capture`.
*/
import { readFileSync, writeFileSync } from 'node:fs';
import { dirname, join } from 'node:path';
import { fileURLToPath } from 'node:url';
import { parseEnrichResponse } from '../src/core/enrich';

interface Fixture {
id: string;
category: string;
raw: string;
}

const here = dirname(fileURLToPath(import.meta.url));
const fixtures = JSON.parse(
readFileSync(join(here, 'dataset', 'capture-fixtures.json'), 'utf8'),
) as Fixture[];

function naiveParses(raw: string): boolean {
try {
const value: unknown = JSON.parse(raw);
return typeof value === 'object' && value !== null && !Array.isArray(value);
} catch {
return false;
}
}

const rows = fixtures.map((fixture) => {
const parsed = parseEnrichResponse(fixture.raw);
return {
naive: naiveParses(fixture.raw),
robust: parsed !== null,
title: Boolean(parsed?.title),
summary: Boolean(parsed?.summary),
tags: (parsed?.tags?.length ?? 0) > 0,
};
});

const total = fixtures.length;
const count = (predicate: (r: (typeof rows)[number]) => boolean): number =>
rows.filter(predicate).length;

const naiveOk = count((r) => r.naive);
const robustOk = count((r) => r.robust);
const robustOnly = count((r) => r.robust && !r.naive);
const withTitle = count((r) => r.title);
const withSummary = count((r) => r.summary);
const withTags = count((r) => r.tags);

const pct = (n: number): string => `${((n / total) * 100).toFixed(1)}%`;

const lines: string[] = [];
lines.push('# OpenRead — Capture Enrichment Eval');
lines.push('');
lines.push(
`Offline, deterministic run over **${total}** small-model reply fixtures. ` +
'No network, no Ollama server needed.',
);
lines.push('');
lines.push('| Metric | Count | Rate |');
lines.push('| --- | --- | --- |');
lines.push(
`| Naive \`JSON.parse\` yields an object | ${naiveOk} | ${pct(naiveOk)} |`,
);
lines.push(
`| Robust \`parseEnrichResponse\` yields usable metadata | ${robustOk} | ${pct(robustOk)} |`,
);
lines.push(`| Usable title recovered | ${withTitle} | ${pct(withTitle)} |`);
lines.push(
`| Usable summary recovered | ${withSummary} | ${pct(withSummary)} |`,
);
lines.push(`| ≥1 tag recovered | ${withTags} | ${pct(withTags)} |`);
lines.push('');
lines.push(
`_The robust parser recovers usable metadata from **${robustOnly}** replies ` +
'that a naive `JSON.parse` drops (fenced, preamble-wrapped, or trailing-prose ' +
'output), while rejecting empty/garbage replies that naive parsing would wave ' +
'through. Enrichment is best-effort — every capture writes a raw note ' +
'regardless._',
);
lines.push('');

const report = lines.join('\n');
console.log(report);
writeFileSync(join(here, 'CAPTURE-RESULTS.md'), report + '\n', 'utf8');
72 changes: 72 additions & 0 deletions eval/dataset/capture-fixtures.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
[
{
"id": "clean-01",
"category": "clean-json",
"raw": "{\"title\":\"線性時間演算法\",\"summary\":\"這段說明演算法以線性時間執行。\",\"tags\":[\"演算法\",\"複雜度\",\"效能\"]}"
},
{
"id": "clean-02",
"category": "clean-json",
"raw": "{\"title\":\"Transformer 架構\",\"summary\":\"介紹自注意力機制。\",\"tags\":[\"nlp\",\"transformer\",\"attention\"]}"
},
{
"id": "fenced-01",
"category": "fenced-json",
"raw": "```json\n{\"title\":\"向量資料庫\",\"summary\":\"說明向量檢索的原理。\",\"tags\":[\"database\",\"embeddings\"]}\n```"
},
{
"id": "fenced-02",
"category": "fenced-json",
"raw": "```\n{\"title\":\"梯度下降\",\"tags\":[\"optimization\",\"training\"]}\n```"
},
{
"id": "preamble-01",
"category": "preamble-json",
"raw": "Sure! Here is the metadata you asked for:\n\n{\"title\":\"知識圖譜\",\"summary\":\"描述實體與關係的建模。\",\"tags\":[\"graph\",\"knowledge\"]}"
},
{
"id": "preamble-02",
"category": "preamble-json",
"raw": "好的,這是整理後的結果:{\"title\":\"檢索增強生成\",\"summary\":\"結合檢索與生成的方法。\",\"tags\":[\"rag\",\"llm\",\"retrieval\"]}"
},
{
"id": "trailing-01",
"category": "trailing-prose",
"raw": "{\"title\":\"擴散模型\",\"summary\":\"逐步去雜訊生成影像。\",\"tags\":[\"diffusion\",\"generative\"]} 希望對你有幫助!"
},
{
"id": "comma-tags-01",
"category": "comma-tags",
"raw": "{\"title\":\"提示工程\",\"summary\":\"設計提示以引導模型。\",\"tags\":\"prompt, engineering, llm\"}"
},
{
"id": "hash-tags-01",
"category": "hash-tags",
"raw": "{\"title\":\"量化\",\"summary\":\"降低模型精度以節省記憶體。\",\"tags\":[\"#Quantization\",\"#Model Size\"]}"
},
{
"id": "partial-01",
"category": "partial",
"raw": "{\"summary\":\"一段關於微調的說明。\"}"
},
{
"id": "refusal-01",
"category": "refusal",
"raw": "I'm sorry, I can't produce that."
},
{
"id": "garbage-01",
"category": "garbage",
"raw": "title: 混亂輸出\ntags - a, b, c"
},
{
"id": "empty-fields-01",
"category": "empty-fields",
"raw": "{\"title\":\"\",\"summary\":\"\",\"tags\":[]}"
},
{
"id": "malformed-01",
"category": "malformed",
"raw": "{\"title\": \"未閉合"
}
]
3 changes: 2 additions & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
"type": "module",
"license": "MIT",
"engines": {
"node": ">=20"
"node": ">=22"
},
"scripts": {
"dev": "wxt",
Expand All @@ -20,6 +20,7 @@
"lint": "eslint .",
"format": "prettier --write .",
"eval": "tsx eval/run.ts",
"eval:capture": "tsx eval/capture-run.ts",
"postinstall": "wxt prepare"
},
"devDependencies": {
Expand Down
9 changes: 4 additions & 5 deletions pnpm-workspace.yaml
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
# Build-script allowlist. esbuild ships a platform binary via a postinstall
# script; WXT/Vite/Vitest all need it, so allow it. spawn-sync is an unused
# prebuilt fallback — explicitly denied so the tooling stops prompting.
# Build-script approval (pnpm 11 `allowBuilds`; replaced v10's
# onlyBuiltDependencies). esbuild ships a platform binary via a postinstall
# script that WXT / Vite / Vitest need, so approve it. spawn-sync is an unused
# prebuilt fallback — explicitly reviewed-and-denied so pnpm stops flagging it.
allowBuilds:
esbuild: true
spawn-sync: false
onlyBuiltDependencies:
- esbuild
47 changes: 46 additions & 1 deletion src/api/ollama.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,12 @@ import { generateSystemPrompt, getFewShotMessages } from '../core/prompt';
import { cleanTranslationOutput } from '../core/sanitize';
import { toTraditionalTW } from '../core/zh-convert';
import { StreamAssembler } from '../core/stream';
import type { ChatMessage, TranslationContext } from '../core/types';
import { buildEnrichMessages, parseEnrichResponse } from '../core/enrich';
import type {
ChatMessage,
TranslationContext,
EnrichResult,
} from '../core/types';

/** Build the OpenAI-compatible chat endpoint for an Ollama server base URL. */
function endpoint(baseUrl: string): string {
Expand Down Expand Up @@ -179,3 +184,43 @@ export async function translateText(params: TranslateParams): Promise<string> {
const cleaned = cleanTranslationOutput(text, content);
return wantsTraditional(targetLang) ? toTraditionalTW(cleaned) : cleaned;
}

export interface EnrichParams {
text: string;
baseUrl: string;
model: string;
targetLang: string;
signal?: AbortSignal;
}

/**
* Single non-streamed enrichment call: ask a small model to label a capture
* with a title, summary, and tags. Best-effort by design — returns null on any
* HTTP or parse failure so a weak local model can never block a capture.
* Temperature 0 for determinism.
*/
export async function enrichText(
params: EnrichParams,
): Promise<EnrichResult | null> {
const { text, baseUrl, model, targetLang, signal } = params;
if (!text || !baseUrl) return null;

const response = await fetch(endpoint(baseUrl), {
method: 'POST',
signal,
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
model,
messages: buildEnrichMessages(text, targetLang),
temperature: 0,
stream: false,
}),
});
if (!response.ok) return null;

const data = (await response.json()) as {
choices?: Array<{ message?: { content?: string } }>;
};
const content = data.choices?.[0]?.message?.content ?? '';
return parseEnrichResponse(content);
}
Loading
Loading