Tune LLM API infra, add benchmark/eval scripts, document in docs/AGENTS.md

docs/AGENTS.md

@@ -0,0 +1,53 @@
+# Agent Notes
+
+Operational knowledge recorded by agents working on this repo. Update as things change.
+
+## LLM API (llm.elimelt.com → Ollama)
+
+### Endpoints
+- Public: `https://llm.elimelt.com` — both Ollama-native (`/api/chat`) and OpenAI-compatible (`/v1/chat/completions`, `/v1/models`) APIs.
+- Internal (from host): `http://172.27.0.11:11434`.
+- Model management endpoints (`/api/pull`, `/api/delete`, `/api/copy`, `/api/push`, `/api/create`) are blocked at the edge by Caddy. Manage models via `docker exec ollama ollama pull|rm <model>`.
+
+### CORS / rate limiting (Caddyfile)
+- Allowed origins: `https://*.elimelt.com` and `http://localhost:5173` (exact spelling — `127.0.0.1` and other ports do NOT match).
+- Requests from allowed origins are exempt from rate limiting; all other traffic is limited to 100 req/min per IP.
+- Gotcha: Ollama validates the `Origin` header itself and returns 403 unless `OLLAMA_ORIGINS=*` is set. Origin policy is enforced by Caddy at the edge instead.
+
+### Ollama container tuning (docker-compose.yml)
+- `OLLAMA_NUM_PARALLEL=4`: concurrency benchmarks (`infra/scripts/bench_ollama_concurrency.py`) showed throughput plateaus around 4 concurrent requests on the i9-13900HK (CPU-bound).
+- `OLLAMA_MAX_QUEUE=20`: rejects excess requests instead of unbounded queueing.
+- `OLLAMA_MAX_LOADED_MODELS=2`: requesting a non-resident model triggers a cold load (up to ~13s for 18GB models) and may evict a warm one.
+- Memory limit 50G: 26B-class models OOM-killed the container at the old 6G limit.
+
+## Model lineup (as of 2026-08-02)
+
+Chosen to be Pareto-optimal per use case. `gemma3:27b` and `llama3.2:3b` were
+deleted as dominated (see benchmarks below).
+
+| Model | Size | Role | Warm 1st content | Decode |
+|---|---|---|---|---|
+| `gemma2:2b` | 1.6GB | speed / TTS-rewrite | 206ms | 27.4 t/s |
+| `gemma4:e4b` | 9.6GB | mid quality, fast cold load (4.4s) | 439ms (`think: false`) | 15.4 t/s |
+| `gemma4:26b` | 18GB | best quality-per-second (MoE, 3.8B active) | 448ms (`think: false`) | 16.7 t/s |
+| `gpt-oss:20b` | 13.8GB | reasoning / tool calling (MoE, 3.6B active) | 1.5s (`think: "low"`) | 12.8 t/s |
+| `qwen2.5-coder:7b` | 4.7GB | code / FIM | 200ms | 11.2 t/s |
+
+### Thinking models — critical gotchas
+- `gemma4:*` think **by default**. Pass `"think": false` on `/api/chat` or first content is delayed ~10s (e4b) to ~37s (26b). With a small `num_predict`, the entire budget can be consumed by thinking and no content ever arrives.
+- `gpt-oss` cannot disable thinking, only effort: `"think": "low" | "medium" | "high"`. Low ≈ 1.5s to first content; high ≈ 28s.
+- The OpenAI `/v1` endpoint has **no way to pass `think`** — gemma4 models think by default there. If a `/v1` client needs gemma4 without thinking, create a Modelfile variant (not yet done; see issues).
+- When benchmarking, distinguish *first token* (may be thinking) from *first content*. `infra/scripts/bench_ollama.py` handles this.
+
+### Benchmark/eval scripts (`infra/scripts/`)
+- `bench_ollama.py <host> [runs]` — TTFT (first token vs first content), decode/prompt t/s, per think-mode.
+- `bench_ollama_concurrency.py` — throughput vs concurrent requests.
+- `bench_ollama_prefill.py` — prompt-processing latency vs input length.
+- `eval_tts_rewrite.py <host> <models>` — quality eval for the notes→TTS rewrite task (math excerpts from notes.elimelt.com, checks for leftover LaTeX/symbols, preambles, dropped terms).
+- `eval_tts_prompts.py` — prompt-variant ablations for the same task.
+
+### Misc findings
+- MoE models (`gemma4:26b`, `gpt-oss:20b`) decode at small-model speeds on CPU despite large total size — only active params matter per token.
+- Small dense models: optimal `num_thread` is 10–14 on the i9-13900HK.
+- `gemma2:2b` passed all automated TTS-rewrite checks but made a semantic error (conflated largest/smallest singular value) that `gemma4:e4b`/`26b` got right. Automated checks don't catch meaning errors.
+- `llama3.2:3b` left raw LaTeX in TTS output; `gemma4` spells code identifiers letter-by-letter ("n p dot ...").

infra/Caddyfile

@@ -25,7 +25,7 @@
 	rate_limit {
 		zone llm_zone {
 			key    {remote_host}
-			events 20
+			events 100
 			window 1m
 		}
 	}
@@ -103,29 +103,53 @@
 	# llm.elimelt.com → Ollama (OpenAI-compatible LLM API)
 	@llm host llm.elimelt.com
 	handle @llm {
-		# Rate limit: 20 requests per minute per IP
-		import rate_limit_llm
+		# Set CORS headers for allowed origins (non-terminal, always runs)
+		@cors_allowed {
+			expression {http.request.header.Origin}.matches("^https://[a-zA-Z0-9-]+\\.elimelt\\.com$") || {http.request.header.Origin} == "http://localhost:5173"
+		}
+		header @cors_allowed Access-Control-Allow-Origin "{http.request.header.Origin}"
+
+		# Handle CORS preflight
+		@cors_preflight method OPTIONS
+		handle @cors_preflight {
+			@cors_allowed_preflight {
+				expression {http.request.header.Origin}.matches("^https://[a-zA-Z0-9-]+\\.elimelt\\.com$") || {http.request.header.Origin} == "http://localhost:5173"
+			}
+			header @cors_allowed_preflight Access-Control-Allow-Methods "GET, POST, OPTIONS"
+			header @cors_allowed_preflight Access-Control-Allow-Headers "Content-Type, Authorization"
+			header @cors_allowed_preflight Access-Control-Max-Age "86400"
+			respond @cors_allowed_preflight 204
+			respond "Forbidden" 403
+		}
 
-		# CORS for *.elimelt.com and localhost
-		import cors_elimelt
+		# Block model management endpoints
+		@blocked path /api/pull /api/delete /api/copy /api/push /api/create
+		handle @blocked {
+			respond "Forbidden" 403
+		}
 
-		# Block model management - pull/delete/copy/push
-		@blocked_pull path /api/pull
-		@blocked_delete path /api/delete
-		@blocked_copy path /api/copy
-		@blocked_push path /api/push
-		@blocked_create path /api/create
-		respond @blocked_pull "Forbidden" 403
-		respond @blocked_delete "Forbidden" 403
-		respond @blocked_copy "Forbidden" 403
-		respond @blocked_push "Forbidden" 403
-		respond @blocked_create "Forbidden" 403
-
-		reverse_proxy ollama:11434 {
-			header_down -Access-Control-Allow-Origin
-			header_down -Access-Control-Allow-Methods
-			header_down -Access-Control-Allow-Headers
-			header_down -Access-Control-Allow-Credentials
+		# Internal origins: no rate limit, proxy to Ollama
+		@internal_origin {
+			expression {http.request.header.Origin}.matches("^https://[a-zA-Z0-9-]+\\.elimelt\\.com$") || {http.request.header.Origin} == "http://localhost:5173"
+		}
+		handle @internal_origin {
+			reverse_proxy ollama:11434 {
+				header_down -Access-Control-Allow-Origin
+				header_down -Access-Control-Allow-Methods
+				header_down -Access-Control-Allow-Headers
+				header_down -Access-Control-Allow-Credentials
+			}
+		}
+
+		# External/no origin: rate limited, proxy to Ollama
+		handle {
+			import rate_limit_llm
+			reverse_proxy ollama:11434 {
+				header_down -Access-Control-Allow-Origin
+				header_down -Access-Control-Allow-Methods
+				header_down -Access-Control-Allow-Headers
+				header_down -Access-Control-Allow-Credentials
+			}
 		}
 	}
 

infra/docker-compose.yml

@@ -262,6 +262,7 @@ services:
   # Ollama - Local LLM inference with OpenAI-compatible API
   # Models are stored in ollama_data volume, download via: docker exec ollama ollama pull <model>
   # API: http://ollama:11434/v1/chat/completions (OpenAI-compatible)
+  # Backpressure: OLLAMA_NUM_PARALLEL controls concurrent inference; excess requests queue.
   ollama:
     image: ollama/ollama:latest
     container_name: ollama
@@ -270,10 +271,23 @@ services:
       - ollama_data:/root/.ollama
     environment:
       - OLLAMA_HOST=0.0.0.0
-      - OLLAMA_NUM_PARALLEL=${OLLAMA_NUM_PARALLEL:-2}
+      # Accept any Origin header; CORS/origin policy is enforced by Caddy at the edge
+      - OLLAMA_ORIGINS=*
+      # Concurrent requests per model (queues excess). Keep low for backpressure.
+      - OLLAMA_NUM_PARALLEL=${OLLAMA_NUM_PARALLEL:-4}
+      # Max models in memory simultaneously
       - OLLAMA_MAX_LOADED_MODELS=${OLLAMA_MAX_LOADED_MODELS:-2}
+      # Queue timeout: reject if queued too long (prevents infinite queue buildup)
+      - OLLAMA_MAX_QUEUE=${OLLAMA_MAX_QUEUE:-20}
     networks:
       - dev_network
+    # Memory limits to prevent OOM
+    deploy:
+      resources:
+        limits:
+          memory: 50G
+        reservations:
+          memory: 2G
     healthcheck:
       test: ["CMD", "curl", "-f", "http://localhost:11434/api/tags"]
       interval: 30s

infra/scripts/bench_ollama.py

@@ -0,0 +1,159 @@
+#!/usr/bin/env python3
+"""Benchmark TTFT and tok/sec for all models on an Ollama server.
+
+Usage: python3 bench_ollama.py [host] [runs]
+  host: Ollama base URL (default http://localhost:11434)
+  runs: measured runs per model after 1 warm-up (default 3)
+
+Stdlib only. Streams /api/chat to measure wall-clock TTFT; decode tok/s
+comes from Ollama's own eval_count/eval_duration stats.
+"""
+import json
+import statistics
+import sys
+import time
+import urllib.request
+
+HOST = sys.argv[1] if len(sys.argv) > 1 else "http://localhost:11434"
+RUNS = int(sys.argv[2]) if len(sys.argv) > 2 else 3
+PROMPT = "Explain what a hash table is in about one paragraph."
+NUM_PREDICT = 100
+
+
+def list_models():
+    with urllib.request.urlopen(f"{HOST}/api/tags", timeout=30) as r:
+        return [m["name"] for m in json.load(r)["models"]]
+
+
+def bench_once(model, think=None):
+    """One streamed request.
+
+    Returns dict with:
+      first_tok_s:     time to first streamed token (thinking or content)
+      first_content_s: time to first content token (None if none arrived)
+      think_tokens:    approx thinking tokens emitted (chunk count)
+      decode_tok_s, prompt_tok_s, eval_count: from Ollama's stats
+    """
+    # Thinking modes need a larger budget so content tokens actually arrive
+    budget = NUM_PREDICT if not think else 600
+    payload = {
+        "model": model,
+        "messages": [{"role": "user", "content": PROMPT}],
+        "stream": True,
+        "options": {"num_predict": budget, "temperature": 0},
+    }
+    if think is not None:
+        payload["think"] = think
+    req = urllib.request.Request(
+        f"{HOST}/api/chat", data=json.dumps(payload).encode(),
+        headers={"Content-Type": "application/json"},
+    )
+    start = time.perf_counter()
+    first_tok = first_content = None
+    think_tokens = 0
+    final = None
+    with urllib.request.urlopen(req, timeout=1800) as r:
+        for line in r:
+            chunk = json.loads(line)
+            msg = chunk.get("message", {})
+            if msg.get("thinking"):
+                think_tokens += 1
+                if first_tok is None:
+                    first_tok = time.perf_counter() - start
+            if msg.get("content"):
+                now = time.perf_counter() - start
+                if first_tok is None:
+                    first_tok = now
+                if first_content is None:
+                    first_content = now
+            if chunk.get("done"):
+                final = chunk
+    return {
+        "first_tok_s": first_tok,
+        "first_content_s": first_content,
+        "think_tokens": think_tokens,
+        "decode_tok_s": final["eval_count"] / (final["eval_duration"] / 1e9),
+        "prompt_tok_s": final["prompt_eval_count"]
+                        / max(final["prompt_eval_duration"] / 1e9, 1e-9),
+        "eval_count": final["eval_count"],
+    }
+
+
+def fmt_ms(seconds):
+    return f"{seconds*1000:.0f}ms" if seconds is not None else "n/a"
+
+
+def bench_model(model, think=None):
+    label = f"{model}" + (f" (think={think})" if think is not None else "")
+    print(f"\n=== {label} ===")
+    # Warm-up: loads model into memory (cold first-token reported separately)
+    try:
+        cold = bench_once(model, think)
+    except Exception as e:
+        print(f"  FAILED to load/run: {e}")
+        return None
+    print(f"  cold first token (incl. model load): {fmt_ms(cold['first_tok_s'])}")
+
+    runs = []
+    for i in range(RUNS):
+        r = bench_once(model, think)
+        runs.append(r)
+        extra = (f"  thinking={r['think_tokens']} tok" if r["think_tokens"] else "")
+        print(f"  run {i+1}: first_tok={fmt_ms(r['first_tok_s'])}  "
+              f"first_content={fmt_ms(r['first_content_s'])}  "
+              f"decode={r['decode_tok_s']:.1f} tok/s  "
+              f"prompt={r['prompt_tok_s']:.1f} tok/s  "
+              f"({r['eval_count']} tokens){extra}")
+
+    def med(key):
+        vals = [r[key] for r in runs if r[key] is not None]
+        return statistics.median(vals) if vals else None
+
+    return {
+        "label": label,
+        "cold_first_tok_s": cold["first_tok_s"],
+        "first_tok_ms": med("first_tok_s") * 1000 if med("first_tok_s") else None,
+        "first_content_ms": (med("first_content_s") * 1000
+                             if med("first_content_s") else None),
+        "decode_tok_s": med("decode_tok_s"),
+        "prompt_tok_s": med("prompt_tok_s"),
+    }
+
+
+# Thinking-capable models get benchmarked in both modes; gpt-oss cannot
+# fully disable thinking, only lower effort.
+THINK_MODES = {
+    "gemma4": [False, True],
+    "gpt-oss": ["low", "high"],
+}
+
+
+def think_modes_for(model):
+    for prefix, modes in THINK_MODES.items():
+        if model.startswith(prefix):
+            return modes
+    return [None]
+
+
+def main():
+    models = list_models()
+    print(f"Ollama @ {HOST} | models: {', '.join(models)} | {RUNS} runs each")
+    results = [r for m in models for t in think_modes_for(m)
+               if (r := bench_model(m, t))]
+
+    def cell(val, fmt):
+        return format(val, fmt) if val is not None else "   n/a"
+
+    print(f"\n{'Model':<28} {'Cold 1st tok':>12} {'1st tok':>9} "
+          f"{'1st content':>12} {'Decode':>10} {'Prompt':>10}")
+    print("-" * 86)
+    for r in results:
+        print(f"{r['label']:<28} {cell(r['cold_first_tok_s'], '>11.2f')}s "
+              f"{cell(r['first_tok_ms'], '>7.0f')}ms "
+              f"{cell(r['first_content_ms'], '>10.0f')}ms "
+              f"{cell(r['decode_tok_s'], '>6.1f')} t/s "
+              f"{cell(r['prompt_tok_s'], '>6.1f')} t/s")
+
+
+if __name__ == "__main__":
+    main()