Spike: the homelab bench — LLM capacity + truth floor on a CCX23-class box (4 vCPU / 16 GB / no GPU), shipped demo profile

[genwave-radio]

🎯 Why

T336 (gh-#438) measured the truth floor on an RTX 4090: every model benched sits above it (worst share 0.3%). But the week logged zero timeouts and zero over-length rejects — a GPU hides the capacity floor entirely. HARDWARE.md still carries exactly one CPU-fence fact (llama3.2:3b on 1 CPU / 6 GB, ~25–30 s warm, observed 2026-07-21).

Home labbers won't devote a GPU box to an internal radio. The number they need is: what fits on 4 cores / 16 GB / no GPU, and is it truthful at that size? Benching on CCX23-class hardware also frees the dev box (no week-long stack freeze) and removes the gh-#610 doubled-library confound.

🔬 Plan

• Fresh throwaway CCX23 via the wizard on v5.4.0 (carries the gh-Thinking-capable models return empty copy via /v1/chat/completions — need a reasoning control #620 reasoning fix). Not the live demo. Doubles as the v5.4.0 fresh-install proof.
• Shipped demo profile as-is: ollama cpus: 1.0 / memory: 6144M, Llm:TimeoutSeconds 60, OLLAMA_KEEP_ALIVE=-1. A model that can't answer within 60 s on one fenced core is below the capacity floor by definition. Wider fences are a later axis (YAGNI).
• Same script + cron as T336 (genwave-T336.sh, on-box against localhost:8080). report gains one line: the capacity share per model — (timeout + emptycompletion + overlength) / copy calls — beside the truth share. No new tooling; the F139 counters already key every cause per model.
• Ladder, 1–2 days each (verify tags before pulling): llama3.2:3b (control) → qwen3:4b (a thinking model at homelab size — the Thinking-capable models return empty copy via /v1/chat/completions — need a reasoning control #620 fix's real test) → gemma3:4b → phi4-mini → then llama3.1:8b and mistral:7b one day each to prove the ceiling.
• Read both floors from the same counters → HARDWARE.md sizing rows + the ollama row get their CPU-fence numbers.

⚠️ Gotchas

• compose.demo.yaml pins ollama/ollama:0.9.6 — too old for gemma4-class tags, and reasoning_effort on the OpenAI-compat layer is unverified there (the Thinking-capable models return empty copy via /v1/chat/completions — need a reasoning control #620 fix was proven on 0.32.5). The bench box needs a newer pin → the pin bump is itself under test; the DEPLOYMENT.md fence-drift grep applies.
• ollama-init pulls only llama3.2:3b: docker exec … ollama pull <tag> before each Llm:Model switch, or that day reads as connectionfailure.
• The box needs media. A 9k-track library ≈ 11 h of enrichment on 4 cores contending with ollama's one — start the bench after enrichment settles, or ship a 1–2k slice.
• The thinking 12–24B class (gemma4:12b, qwen3:14b, deepseek-r1:14b, magistral:24b) does not fit this shape (magistral doesn't fit 16 GB at all). gemma4:12b's own verdict — the feat: Better LLM observability #365 flapper — is a GPU-owner question: a 1–2 day 4090 spot-check any time after Thinking-capable models return empty copy via /v1/chat/completions — need a reasoning control #620 ships, no freeze needed.

📦 Deliverables

• HARDWARE.md: the capacity-floor table (model × timeout share × truth share) on the shipped shape, benched build + dates; sizing rows updated.
• Close-out note here and on spike: patter names the wrong day/daypart against an explicit clock line — model size vs family vs prompt shape #438.

Sequencing: after the v5.4.0 tag — #610 cleanup → v5.4.0 canary on dev → tag → wizard the CCX23 → bench.

Refs: #438 (T336 truth floor), #620 (thinking models), #365 (the gemma flap history), #277 (length-compliance family), #614 (the TTS sibling with the same harness idea).

[genwave-radio]

📌 Verified today while building #630 (the #620 fix): the demo's pinned ollama/ollama:0.9.6 answers HTTP 200 with reasoning_effort present (omitted / none / high all fine on a throwaway container) — the field is ignored, not rejected, so v5.4.0 cannot go red on the demo. It is still too old for gemma4-class tags, so the bench box's pin bump stands; drop the "reasoning_effort unverified there" gotcha.