Make the local embedding dtype configurable for practical multilingual recall

[mcthesw]

Short description

Allow embedding.dtype: "q8" for local Transformers.js models instead of hard-coding fp32.

What problem does this solve?

My Pi conversations are mostly Chinese, while historian summaries, memories, Git commits, and code identifiers often contain English.

The default Xenova/all-MiniLM-L6-v2 model is very lightweight, but in a small retrieval test based on this workload it performed poorly when matching Chinese queries to English memories.

I tested 17 manually labeled retrieval cases using the same invocation as Magic Context:

• Transformers.js feature-extraction
• mean pooling
• normalized embeddings
• no query/document prompts

Results:

Model	dtype	Top-1	Top-3	Chinese → English Top-1	Peak RSS
Xenova/all-MiniLM-L6-v2	fp32	47.1%	47.1%	0/5	~380 MiB
Xenova/paraphrase-multilingual-MiniLM-L12-v2	fp32	94.1%	100%	4/5	~1430 MiB
Xenova/paraphrase-multilingual-MiniLM-L12-v2	q8	94.1%	100%	4/5	~858 MiB

The multilingual model solves the recall problem, but fp32 is expensive for a coding-agent process that may also run parallel subagents.

The q8 version preserved the retrieval results in this test while reducing the ONNX weight from about 448 MiB to 113 MiB and reducing peak RSS substantially.

Magic Context 0.33.0 currently hard-codes:

pipeline("feature-extraction", this.model, {
   dtype: "fp32"
})

So users cannot select the quantized model through configuration.

Proposed solution

Add an optional dtype setting for the local embedding provider like:

{
  "embedding": {
    "provider": "local",
    "model": "Xenova/paraphrase-multilingual-MiniLM-L12-v2",
    "dtype": "q8"
  }
}

Alternatives considered

• Keep all-MiniLM-L6-v2: lightweight, but Chinese-to-English recall was unreliable in this workload
• m3e-small: lightweight (~95 MiB ONNX), but only reached 58.8% Top-1 and degraded English retrieval in the same test.
• multilingual-e5-small: requires query/passage prefixes. The local provider currently accepts a purpose argument but does not use it to format model input.

Area

Pi plugin

Additional context

my env:

• Magic Context 0.33.0
• Pi 0.82.1
• Node.js 24.14.0
• macOS arm64

[mcthesw]

Related schema inconsistency:

The schema says model is ignored for local embeddings:

magic-context/packages/plugin/src/config/schema/magic-context.ts

Line 259 in 113f3e4

.describe("Embedding model name. Required for openai-compatible, ignored for local."),

But the local config preserves the selected model:

magic-context/packages/plugin/src/config/schema/magic-context.ts

Lines 333 to 338 in 113f3e4

	if (data.provider === "local") {
	return {
	provider: "local" as const,
	model: data.model?.trim() || DEFAULT_LOCAL_EMBEDDING_MODEL,
	...(data.max_input_tokens ? { max_input_tokens: data.max_input_tokens } : {}),
	};

magic-context/packages/plugin/src/features/magic-context/memory/embedding.ts

Lines 120 to 122 in 113f3e4

	if (config.provider === "local") {
	return new LocalEmbeddingProvider(config.model, config.max_input_tokens);
	}

magic-context/packages/plugin/src/features/magic-context/memory/embedding-local.ts

Lines 513 to 517 in 113f3e4

	const pipeline = await withQuietConsole(() =>
	createPipeline("feature-extraction", this.model, {
	dtype: "fp32",
	...(injectedWasmOrt ? { device: "auto" } : {}),
	}),

So local model selection is supported; only the schema description appears outdated.