diff --git a/CHANGELOG.md b/CHANGELOG.md index 02eb0637..7a56a275 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,7 @@ Format: version sections are listed newest first. ## [Unreleased] ### Added +- **q27 LLM backend** — detect and parse the signalnine/q27 engine (`/v1/models` `owned_by: "q27"` or Prometheus `q27_*` series) instead of mislabeling it as vLLM. Live tok/s from `q27_*_processed` counter diffs (real-time during generation; completion-based per-api totals as fallback for older binaries), exact computed-only prefill whose cached/uncached split doubles as the prefix-cache hit rate, KV usage, slots, TTFT/E2E/ITL p95 histograms, MTP accept, engine Active state, and constant-0 preemptions (FIFO admission — no wait queue, the Requests tile reads “N run”). Inference-health tiles render for q27 like vLLM. DecodeBench server-side tok/s reads the same q27 counters. - **DGX Spark CPU temperature** — remote Sparks collect CPU temp over SSH with the same hwmon allowlist as hosts (`acpitz` / `coretemp` / `k10temp` / `zenpower`; NVMe / CX7 filtered out). Overview shows a CPU bar and Spark pages show a CPU row on the GPU panel when the reading is above 0°C. ### Fixed diff --git a/README.md b/README.md index 2b8b300a..23639a09 100644 --- a/README.md +++ b/README.md @@ -67,13 +67,13 @@ Full history: [CHANGELOG.md](./CHANGELOG.md) | **Non-Spark GPU hosts** | Linux boxes with a dedicated NVIDIA GPU are first-class units: same `nvidia-smi` collectors over SSH, detected hardware summary, and separate **RAM** / **VRAM** panels. Detail page: GPU (left) + **RAM → Network → Storage** (right column); Overview cards show RAM and VRAM bars | | **Live streaming** | WebSocket metrics with configurable poll intervals; central history store for sparklines across tab switches | | **Local + remote** | Host metrics via sysfs/proc/`nvidia-smi`; remotes over SSH (key or password) | -| **LLM probe** | Auto-detects llama.cpp, vLLM, sglang, ds4-server, or EXL3; live decode/prefill tok/s; cached vs uncached prefill on ds4, llama.cpp, and SGLang; **daily peak** history on the LLM card | +| **LLM probe** | Auto-detects llama.cpp, vLLM, sglang, ds4-server, EXL3, or q27; live decode/prefill tok/s; cached vs uncached prefill on ds4, llama.cpp, SGLang, and q27; **daily peak** history on the LLM card | | **ComfyUI** | Opt-in probe: queue/jobs, progress, cancel, Open link, inventory, overview chip | | **Hermes Agent** | Opt-in per unit: background update check (10 min), status badges, one-click or batch `hermes update` | | **Tailnet** | Opt-in probe: flags a unit that is healthy on the LAN but off its tailnet | | **Decode benchmark** | Multi-concurrency streaming decode tok/s; type picker (Structured / Prose / Code / JSON); lab protocol (temp 0, thinking off); persisted last run | | **Prompt Showcase** | Full-page multi-terminal LLM streaming demo (up to 32 prompts) with live tok/s and copy-out | -| **vLLM health** | KV cache %, run/wait queue, TTFT/E2E/ITL p95, preemptions, prefix cache, MTP accept from Prometheus `/metrics` | +| **LLM inference health** | KV cache %, run/wait queue, TTFT/E2E/ITL p95, preemptions, prefix cache, MTP accept from Prometheus `/metrics` (vLLM and q27; q27 FIFO-queues so the Requests tile reads “N run” without a wait gauge) | | **Multiple LLM ports** | Monitor several LLM servers on different ports simultaneously — each gets its own panel with independent backend detection and metrics | | **GPU processes** | See the top GPU processes by VRAM usage directly in the GPU panel, including process name and memory allocation | | **Spark uptime** | System uptime displayed inline on each Spark header for at-a-glance availability | @@ -495,6 +495,8 @@ Each configured LLM port gets its own `LlmProbe` instance running in parallel. P - **llama.cpp** — `/slots` for live decode rates; model from `/props` - **ds4-server** (Entrpi/ds4-on-spark) — `/v1/models` (`owned_by: ds4.c`) + Prometheus `ds4_*` token counters for live tok/s +- **EXL3** (ExLlamaV3 `tools/serve_openai.py`) — `/v1/models` (`owned_by: exl3`) or `/health` `{ok, busy}`; live tok/s from `/health` cumulative counters +- **q27** (signalnine/q27 engine) — `/v1/models` (`owned_by: q27`) or Prometheus `q27_*` series; live tok/s from `q27_*_processed` counter diffs (completion-based totals as fallback), exact computed-only prefill with the cached/uncached split doubling as the prefix-cache hit rate, TTFT/E2E/ITL p95 histograms, and constant-0 preemptions (FIFO admission, no wait queue) - **vLLM / sglang** — `/v1/models`; sglang via `/server_info` (`last_gen_throughput` when metrics off; `/get_server_info` fallback), vLLM via Prometheus `/metrics` counters (scientific notation supported) Rates are derived from per-probe cumulative counter diffs (or SGLang sticky throughput while it moves). Multiple ports can be added or removed at runtime without restarting the monitor. diff --git a/server/collectors/LlmProbe.js b/server/collectors/LlmProbe.js index 08522b2a..0e72caa0 100644 --- a/server/collectors/LlmProbe.js +++ b/server/collectors/LlmProbe.js @@ -66,7 +66,7 @@ export class LlmProbe { this.baseUrl = `http://${llmProbeHost(spark)}:${port}`; // State - this.backendType = null; // 'vllm' | 'llama.cpp' | 'sglang' | 'ds4' | 'exl3' | null + this.backendType = null; // 'vllm' | 'llama.cpp' | 'sglang' | 'ds4' | 'exl3' | 'q27' | null this.serverIsOpenAI = null; // true = OpenAI-compatible /** Whether /v1/models (or /slots) answered without credentials. null = unknown. */ this.authOpen = null; @@ -279,7 +279,8 @@ export class LlmProbe { this.backendType !== "vllm" && this.backendType !== "sglang" && this.backendType !== "ds4" && - this.backendType !== "exl3" + this.backendType !== "exl3" && + this.backendType !== "q27" ) { const slotUrl = `${this.baseUrl}/slots`; try { @@ -326,19 +327,22 @@ export class LlmProbe { } /** - * Classify an OpenAI-compatible server: ds4, SGLang, EXL3, or vLLM (default). + * Classify an OpenAI-compatible server: ds4, SGLang, EXL3, q27, or vLLM (default). * @param {unknown} ownedBy - * @returns {Promise<"ds4" | "sglang" | "exl3" | "vllm">} + * @returns {Promise<"ds4" | "sglang" | "exl3" | "q27" | "vllm">} */ async _classifyOpenAIBackend(ownedBy) { if (typeof ownedBy === "string") { if (/ds4/i.test(ownedBy)) return "ds4"; if (/sglang/i.test(ownedBy)) return "sglang"; if (/exl3/i.test(ownedBy)) return "exl3"; + // q27's /v1/models reports owned_by: "q27" (signalnine/q27 engine). + if (/q27/i.test(ownedBy)) return "q27"; } if (await this._probeIsDs4()) return "ds4"; if (await this._probeIsSglang()) return "sglang"; if (await this._probeIsExl3()) return "exl3"; + if (await this._probeIsQ27()) return "q27"; return "vllm"; } @@ -406,6 +410,25 @@ export class LlmProbe { return /(?:^|\n)ds4_tokens_decoded_total(?:\{|\s)/m.test(String(body || "")); } + /** True when Prometheus /metrics exposes q27-series (signalnine/q27 engine). */ + async _probeIsQ27() { + try { + const res = await this._fetch(`${this.baseUrl}/metrics`); + if (!res.ok) return false; + const txt = await res.text(); + return LlmProbe._metricsLookLikeQ27(txt); + } catch { + return false; + } + } + + /** @param {string} body */ + static _metricsLookLikeQ27(body) { + return /(?:^|\n)q27_(?:decode_tokens_(?:processed_)?total|requests_total)(?:\{|\s)/m.test( + String(body || "") + ); + } + // ─── OpenAI-compatible path (vLLM/sglang/ds4) ──────────── async _probeOpenAICompatible() { const now = Date.now(); @@ -512,15 +535,31 @@ export class LlmProbe { ) { this.backendType = "ds4"; this._applyDs4Metrics(txt, dtSec); + } else if ( + this.backendType === "q27" || + LlmProbe._metricsLookLikeQ27(txt) + ) { + this.backendType = "q27"; + this._applyQ27Metrics(txt, dtSec); } else { this.backendType = "vllm"; this._applyVllmMetrics(txt, dtSec); } - } else if (this.backendType !== "ds4" && this.backendType !== "exl3") { + } else if ( + this.backendType !== "ds4" && + this.backendType !== "exl3" && + this.backendType !== "q27" + ) { this.backendType = "vllm"; } } catch { - if (this.backendType !== "ds4" && this.backendType !== "exl3") this.backendType = "vllm"; + if ( + this.backendType !== "ds4" && + this.backendType !== "exl3" && + this.backendType !== "q27" + ) { + this.backendType = "vllm"; + } } return this._getSnapshot(); @@ -613,6 +652,108 @@ export class LlmProbe { this.itlP95Seconds = null; } + /** + * Apply q27 (signalnine/q27 engine) Prometheus /metrics. + * + * The engine exposes the [req]-universe telemetry that was previously only + * in stderr logs, under q27_* series with an api= label (chat / completions / + * messages / responses). The probe sums across label sets, exactly like the + * ds4/vLLM paths: live tok/s from counter deltas so idle → 0. + * + * Semantics vs the vLLM path: prefill accounting is EXACT (per-request + * token counts, not vLLM's estimates) and the prefix split (computed/cached) + * doubles as the prefix-cache hit rate; the main prefill tile follows the + * ds4 convention and counts COMPUTED tokens only. The waiting tile stays + * null → the panel shows "—" (q27 FIFO-queues, no scheduler wait), while + * preemptions are exposed as a constant-0 counter so Preempts reads 0. + * @param {string} txt + * @param {number} dtSec + */ + _applyQ27Metrics(txt, dtSec) { + // Live processed counters first (move during generation -> real-time + // tok/s); fall back to the completion-based per-api totals for older + // q27 binaries (step function: 0 during generation, jump at completion). + const decoded = + this._getPromMetric(txt, "q27_decode_tokens_processed_total") ?? + this._getPromMetric(txt, "q27_decode_tokens_total"); + // Exact prefill (not estimated like vLLM). Follow the ds4 convention: + // the main prefill tile counts COMPUTED tokens only -- cache-served + // tokens go to the cached/uncached split below, so a cache hit does not + // inflate the "real work" rate. + const computed = + this._getPromMetric(txt, "q27_prefill_computed_tokens_processed_total") ?? + this._getPromMetric(txt, "q27_prefill_computed_tokens_total"); + if (decoded != null) { + if (dtSec > 0 && dtSec < 10) { + const deltaOut = decoded - this.lastTokenCounts.output; + this.generationTps = Math.max(0, Math.round((deltaOut / dtSec) * 100) / 100); + if (computed != null) { + const deltaIn = computed - this.lastTokenCounts.input; + this.prefillTps = Math.max(0, Math.round((deltaIn / dtSec) * 100) / 100); + } + } + if (computed != null) this.lastTokenCounts.input = computed; + this.lastTokenCounts.output = decoded; + this.totalOutputTokens = decoded; + } + + const inflight = this._getPromMetric(txt, "q27_requests_inflight"); + this.requestsRunning = inflight; + if (inflight != null) this.slotsActive = Math.round(inflight); + + const slotsTotal = this._getPromMetric(txt, "q27_slots_total"); + if (slotsTotal != null) this.slotsTotal = Math.round(slotsTotal); + + this.kvCacheUsage = this._getPromMetric(txt, "q27_kv_usage_perc"); + this.requestsWaiting = null; // not exposed: q27 FIFO-queues, no wait gauge + // q27 never preempts (FIFO admission) — the server exposes a constant-0 + // counter, so the Preempts tile reads 0 instead of "—". + this.preemptionsTotal = this._getPromMetric(txt, "q27_preemptions_total"); + // Engine state: q27 keeps weights resident and is ready whenever the + // server is up (no sleep state / memory release), so report Active like + // the SGLang path does. + if (this.gpuMemoryUtilization == null) this.gpuMemoryUtilization = 1; + + // Histograms (cumulative buckets, +Inf == _count by construction). + const ttftHist = this._parseHistogram( + txt, + "q27_ttft_seconds", + "q27_ttft_seconds_count" + ); + const ttftP95 = this._histogramQuantile(ttftHist.buckets, ttftHist.total, 0.95); + this.ttftP95Seconds = ttftP95 == null ? null : Math.round(ttftP95 * 1000) / 1000; + + const e2eHist = this._parseHistogram( + txt, + "q27_e2e_seconds", + "q27_e2e_seconds_count" + ); + const e2eP95 = this._histogramQuantile(e2eHist.buckets, e2eHist.total, 0.95); + this.e2eP95Seconds = e2eP95 == null ? null : Math.round(e2eP95 * 1000) / 1000; + + const itlHist = this._parseHistogram( + txt, + "q27_itl_seconds", + "q27_itl_seconds_count" + ); + const itlP95 = this._histogramQuantile(itlHist.buckets, itlHist.total, 0.95); + this.itlP95Seconds = itlP95 == null ? null : Math.round(itlP95 * 1000) / 1000; + + // Prefix-cache hit rate + live cached/uncached prefill split (live + // processed counters, with completion-based fallback). + const cachedSplit = + this._getPromMetric(txt, "q27_prefill_cached_tokens_processed_total") ?? + this._getPromMetric(txt, "q27_prefill_cached_tokens_total"); + const computedSplit = + this._getPromMetric(txt, "q27_prefill_computed_tokens_processed_total") ?? + this._getPromMetric(txt, "q27_prefill_computed_tokens_total"); + this._setPrefillSplitRates(cachedSplit, computedSplit, dtSec); + + const specAccept = this._getPromMetric(txt, "q27_spec_accept_ratio"); + this.mtpAcceptanceRate = + specAccept != null ? Math.round(specAccept * 10000) / 10000 : null; + } + /** * Apply EXL3 tools/serve_openai.py GET /health. * Live tok/s from cumulative counter diffs so idle → 0. @@ -1212,11 +1353,12 @@ export class LlmProbe { } /** - * Parse a vLLM Prometheus histogram from /metrics text. - * Returns { buckets: [{upper, count}], total } with cumulative counts per `le`, - * summed across label sets. `total` is the summed `_count` series (or null). + * Parse a Prometheus histogram from /metrics text. + * Returns { buckets: [{upper, count}], total } with cumulative counts per + * `le`, summed across label sets. `total` is the summed `_count` series + * (or null). `countMetricName` is the full metric name (with prefix). */ - _parseVllmHistogram(body, metricPrefix) { + _parseHistogram(body, metricPrefix, countMetricName) { const esc = metricPrefix.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); // Bucket lines: _bucket{...le="X"...} VALUE const bucketRe = new RegExp( @@ -1235,7 +1377,7 @@ export class LlmProbe { if (upper === Infinity) infCount += count; byUpper.set(upper, (byUpper.get(upper) || 0) + count); } - const total = this._getVllmMetric(body, `${metricPrefix.replace(/^vllm:/, "")}_count`); + const total = this._getPromMetric(body, countMetricName); // Prometheus invariant: +Inf bucket count == _count. Mismatch → refuse quantile. if (total != null && infCount > 0 && Math.abs(infCount - total) > 1e-6) { return { buckets: [], total: null }; @@ -1245,6 +1387,14 @@ export class LlmProbe { return { buckets, total }; } + /** + * Parse a vLLM Prometheus histogram from /metrics text (vllm: prefix). + */ + _parseVllmHistogram(body, metricPrefix) { + const name = metricPrefix.replace(/^vllm:/, ""); + return this._parseHistogram(body, metricPrefix, `vllm:${name}_count`); + } + /** * Prometheus-style linear interpolation for a histogram quantile. * Returns null when empty / invalid or target is in the +Inf tail. diff --git a/server/collectors/LlmStreaming.js b/server/collectors/LlmStreaming.js index 144acc48..96eb8edc 100644 --- a/server/collectors/LlmStreaming.js +++ b/server/collectors/LlmStreaming.js @@ -94,6 +94,16 @@ export async function readServerGenerationTokens(baseUrl, opts = {}) { /^sglang_generation_tokens_total(?:\{[^}]*\})?\s+([\d.eE+-]+)\s*$/gm ); if (sglang != null) return sglang; + // q27 (signalnine/q27 engine) — live processed counter first, then the + // completion-based per-api total (same preference as LlmProbe). + const q27 = + fromSeries( + /^q27_decode_tokens_processed_total(?:\{[^}]*\})?\s+([\d.eE+-]+)\s*$/gm + ) ?? + fromSeries( + /^q27_decode_tokens_total(?:\{[^}]*\})?\s+([\d.eE+-]+)\s*$/gm + ); + if (q27 != null) return q27; } } catch { /* try next */ diff --git a/server/collectors/SystemCollector.js b/server/collectors/SystemCollector.js index 9ac3a518..d53109ae 100644 --- a/server/collectors/SystemCollector.js +++ b/server/collectors/SystemCollector.js @@ -561,7 +561,7 @@ export class SystemCollector { const lines = output.trim().split("\n").filter(Boolean); const disks = []; const disabledDevices = this.spark.disabledDevices || []; - const PSEUDO = new Set(["tmpfs", "devtmpfs", "proc", "sysfs", "efivarfs", "squashfs", "overlay", "devpts", "cgroup", "cgroup2"]); + const PSEUDO = new Set(["tmpfs", "devtmpfs", "proc", "sysfs", "efivarfs", "squashfs", "overlay", "devpts", "cgroup", "cgroup2", "swap"]); for (const line of lines) { const nameMatch = line.match(/NAME="([^"]*)"/); @@ -1448,7 +1448,10 @@ export class SystemCollector { }); } } - return this._readHostFile(`/proc/net/${relPath}`); + // No host bind mount: read straight from the local /proc/net. + // Do NOT re-dispatch through _readHostFile here — it routes /proc/net/* + // back into this method, causing infinite recursion. + return fs.readFileSync(`/proc/net/${relPath}`, "utf-8"); } /** Lightweight liveness for local Sparks. */ diff --git a/server/collectors/__tests__/LlmProbe.q27.test.js b/server/collectors/__tests__/LlmProbe.q27.test.js new file mode 100644 index 00000000..4236ea4f --- /dev/null +++ b/server/collectors/__tests__/LlmProbe.q27.test.js @@ -0,0 +1,261 @@ +/** + * Unit tests for q27 (signalnine/q27 engine) detection and metrics. + */ +import { test } from "node:test"; +import { strict as assert } from "node:assert"; +import { LlmProbe } from "../LlmProbe.js"; +import { readServerGenerationTokens } from "../LlmStreaming.js"; + +// Realistic q27 /metrics exposition: q27_* series, api= labels (the probe +// sums across label sets). +Inf == _count by construction. +const Q27_METRICS = `# TYPE q27_requests_total counter +q27_requests_total{api="chat"} 3 +q27_requests_total{api="messages"} 1 +# TYPE q27_requests_errors_total counter +q27_requests_errors_total{api="chat"} 0 +# TYPE q27_prompt_tokens_total counter +q27_prompt_tokens_total{api="chat"} 200 +q27_prompt_tokens_total{api="messages"} 50 +# TYPE q27_prefill_computed_tokens_total counter +q27_prefill_computed_tokens_total{api="chat"} 150 +q27_prefill_computed_tokens_total{api="messages"} 50 +# TYPE q27_prefill_cached_tokens_total counter +q27_prefill_cached_tokens_total{api="chat"} 50 +# TYPE q27_decode_tokens_total counter +q27_decode_tokens_total{api="chat"} 400 +q27_decode_tokens_total{api="messages"} 100 +# TYPE q27_requests_inflight gauge +q27_requests_inflight 2 +# TYPE q27_slots_total gauge +q27_slots_total 4 +# TYPE q27_kv_usage_perc gauge +q27_kv_usage_perc 0.42 +# TYPE q27_spec_accept_ratio gauge +q27_spec_accept_ratio 0.87 +# TYPE q27_preemptions_total counter +q27_preemptions_total 0 +# TYPE q27_ttft_seconds histogram +q27_ttft_seconds_bucket{api="chat",le="0.010"} 0 +q27_ttft_seconds_bucket{api="chat",le="0.050"} 0 +q27_ttft_seconds_bucket{api="chat",le="0.100"} 0 +q27_ttft_seconds_bucket{api="chat",le="0.250"} 0 +q27_ttft_seconds_bucket{api="chat",le="0.500"} 2 +q27_ttft_seconds_bucket{api="chat",le="1.000"} 2 +q27_ttft_seconds_bucket{api="chat",le="2.500"} 2 +q27_ttft_seconds_bucket{api="chat",le="5.000"} 2 +q27_ttft_seconds_bucket{api="chat",le="+Inf"} 2 +q27_ttft_seconds_sum{api="chat"} 0.9 +q27_ttft_seconds_count{api="chat"} 2 +`; + +test("_metricsLookLikeQ27: true for q27 exposition", () => { + assert.equal(LlmProbe._metricsLookLikeQ27(Q27_METRICS), true); +}); + +test("_metricsLookLikeQ27: false for vLLM exposition", () => { + assert.equal( + LlmProbe._metricsLookLikeQ27("vllm:generation_tokens_total 10.0\n"), + false + ); +}); + +test("_detectServerType: owned_by q27 → q27", async () => { + const probe = new LlmProbe({ lanIp: "127.0.0.1" }, 8888); + probe._fetch = async (url) => { + const u = String(url); + if (u.endsWith("/slots")) { + return { ok: false, status: 404, json: async () => ({}) }; + } + if (u.endsWith("/v1/models")) { + return { + ok: true, + status: 200, + json: async () => ({ + data: [{ id: "qwen38-27b-mtp", owned_by: "q27", max_model_len: 262144 }], + }), + }; + } + return { ok: false, status: 404, json: async () => ({}) }; + }; + await probe._detectServerType(); + assert.equal(probe.serverIsOpenAI, true); + assert.equal(probe.backendType, "q27"); +}); + +test("_detectServerType: OpenAI models + q27 /metrics → q27", async () => { + const probe = new LlmProbe({ lanIp: "127.0.0.1" }, 8888); + probe._fetch = async (url) => { + const u = String(url); + if (u.endsWith("/slots")) { + return { ok: false, status: 404, json: async () => ({}) }; + } + if (u.endsWith("/v1/models")) { + return { + ok: true, + status: 200, + json: async () => ({ + data: [{ id: "qwen38-27b-mtp" }], + }), + }; + } + if (u.endsWith("/metrics")) { + return { ok: true, status: 200, text: async () => Q27_METRICS }; + } + return { ok: false, status: 404, json: async () => ({}) }; + }; + await probe._detectServerType(); + assert.equal(probe.backendType, "q27"); +}); + +test("_applyQ27Metrics: gauges + counters + split + histograms", () => { + const probe = new LlmProbe({ lanIp: "127.0.0.1" }, 8888); + // First sample seeds counters (no rate yet — needs a prior baseline) + probe._applyQ27Metrics(Q27_METRICS, 2); + assert.equal(probe.totalOutputTokens, 500); // chat 400 + messages 100 + assert.equal(probe.slotsActive, 2); + assert.equal(probe.slotsTotal, 4); + assert.equal(probe.kvCacheUsage, 0.42); + assert.equal(probe.mtpAcceptanceRate, 0.87); + assert.equal(probe.preemptionsTotal, 0); + assert.equal(probe.gpuMemoryUtilization, 1); // weights resident → Active + assert.equal(probe.prefixCacheHitRate, 0.2); // cached 50 / (50 + computed 200) + // Histograms parsed (all TTFT observations in the le=0.5 bucket → p95 ≈ 0.488) + assert.ok( + probe.ttftP95Seconds != null && probe.ttftP95Seconds > 0 && probe.ttftP95Seconds <= 0.5, + `ttft p95=${probe.ttftP95Seconds}` + ); + + // Second sample with same counters → idle → 0 tok/s + probe._applyQ27Metrics(Q27_METRICS, 2); + assert.equal(probe.generationTps, 0); + assert.equal(probe.prefillTps, 0); + + // Counter advanced → live rate from Δ / Δt (computed-only prefill, ds4-style) + const active = Q27_METRICS + .replace("q27_decode_tokens_total{api=\"chat\"} 400", "q27_decode_tokens_total{api=\"chat\"} 550") + .replace( + "q27_prefill_computed_tokens_total{api=\"chat\"} 150", + "q27_prefill_computed_tokens_total{api=\"chat\"} 250" + ) + .replace( + "q27_prefill_cached_tokens_total{api=\"chat\"} 50", + "q27_prefill_cached_tokens_total{api=\"chat\"} 60" + ); + probe._applyQ27Metrics(active, 2); + assert.equal(probe.generationTps, 75); // (550-400)/2 + assert.equal(probe.prefillTps, 50); // computed (250-150)/2 + assert.equal(probe.uncachedPrefillTps, 50); + assert.equal(probe.cachedPrefillTps, 5); // (60-50)/2 + + // Cached tokens jumping must not inflate the main prefill tile + const cachedJump = active.replace( + "q27_prefill_cached_tokens_total{api=\"chat\"} 60", + "q27_prefill_cached_tokens_total{api=\"chat\"} 600" + ); + probe._applyQ27Metrics(cachedJump, 2); + assert.equal(probe.prefillTps, 0); + assert.equal(probe.uncachedPrefillTps, 0); + assert.equal(probe.cachedPrefillTps, 270); // (600-60)/2 +}); + +// Live processed counters: move DURING generation, so tok/s is real-time +// (no completion-time step). No api= labels on these series. +const Q27_LIVE_METRICS = `# TYPE q27_decode_tokens_processed_total counter +q27_decode_tokens_processed_total 400 +# TYPE q27_prefill_computed_tokens_processed_total counter +q27_prefill_computed_tokens_processed_total 150 +# TYPE q27_prefill_cached_tokens_processed_total counter +q27_prefill_cached_tokens_processed_total 50 +# TYPE q27_requests_inflight gauge +q27_requests_inflight 1 +`; + +test("_applyQ27Metrics: live processed counters drive real-time rates", () => { + const probe = new LlmProbe({ lanIp: "127.0.0.1" }, 8888); + probe.lastTokenCounts = { input: 150, output: 400 }; // seeded baseline + probe._applyQ27Metrics(Q27_LIVE_METRICS, 2); + assert.equal(probe.totalOutputTokens, 400); + assert.equal(probe.generationTps, 0); // first sample seeds the baseline + assert.equal(probe.requestsRunning, 1); + + const advanced = Q27_LIVE_METRICS + .replace( + "q27_decode_tokens_processed_total 400", + "q27_decode_tokens_processed_total 620" + ) + .replace( + "q27_prefill_computed_tokens_processed_total 150", + "q27_prefill_computed_tokens_processed_total 190" + ) + .replace( + "q27_prefill_cached_tokens_processed_total 50", + "q27_prefill_cached_tokens_processed_total 60" + ); + probe._applyQ27Metrics(advanced, 2); + assert.equal(probe.generationTps, 110); // (620-400)/2 + assert.equal(probe.prefillTps, 20); // computed (190-150)/2 + assert.equal(probe.uncachedPrefillTps, 20); + assert.equal(probe.cachedPrefillTps, 5); // (60-50)/2 + assert.equal(probe.prefixCacheHitRate, 0.24); // 60/(60+190) +}); + +test("_applyQ27Metrics: +Inf != _count refuses the quantile", () => { + const probe = new LlmProbe({ lanIp: "127.0.0.1" }, 8888); + const broken = Q27_METRICS.replace( + "q27_ttft_seconds_count{api=\"chat\"} 2", + "q27_ttft_seconds_count{api=\"chat\"} 1" + ); + probe._applyQ27Metrics(broken, 2); + assert.equal(probe.ttftP95Seconds, null); +}); + +test("_metricsLookLikeQ27: true for live processed-counters-only exposition", () => { + assert.equal(LlmProbe._metricsLookLikeQ27(Q27_LIVE_METRICS), true); +}); + +test("readServerGenerationTokens: q27 live processed counter", async () => { + const originalFetch = globalThis.fetch; + globalThis.fetch = async (url) => { + if (String(url).endsWith("/metrics")) { + return { ok: true, status: 200, text: async () => Q27_LIVE_METRICS }; + } + return { ok: false, status: 404 }; + }; + try { + const v = await readServerGenerationTokens("http://127.0.0.1:8888"); + assert.equal(v, 400); // q27_decode_tokens_processed_total + } finally { + globalThis.fetch = originalFetch; + } +}); + +test("probe: q27 path does not mislabel as vllm", async () => { + const probe = new LlmProbe({ lanIp: "127.0.0.1" }, 8888); + probe.serverIsOpenAI = true; + probe.backendType = "q27"; + probe.authOpen = true; + probe._lastDetectAt = Date.now(); + probe.lastProbeTime = Date.now() - 2000; + probe.lastTokenCounts = { input: 200, output: 500 }; + probe._fetch = async (url) => { + const u = String(url); + if (u.endsWith("/v1/models")) { + return { + ok: true, + status: 200, + json: async () => ({ + data: [{ id: "qwen38-27b-mtp", owned_by: "q27", max_model_len: 262144 }], + }), + }; + } + if (u.endsWith("/metrics")) { + return { ok: true, status: 200, text: async () => Q27_METRICS }; + } + return { ok: false, status: 404, json: async () => ({}) }; + }; + const snap = await probe.probe(); + assert.equal(snap.backend, "q27"); + assert.equal(snap.available, true); + assert.equal(snap.generationTps, 0); // no delta vs seeded baseline + assert.equal(snap.contextLength, 262144); // from /v1/models max_model_len +}); diff --git a/src/api/types.ts b/src/api/types.ts index 524e030d..be81c556 100644 --- a/src/api/types.ts +++ b/src/api/types.ts @@ -281,7 +281,7 @@ export interface UnifiedMemoryMetrics { // ─── LLM metrics ───────────────────────────────────────── export interface LlmMetrics { available: boolean; - backend: "vllm" | "llama.cpp" | "sglang" | "ds4" | "exl3" | null; + backend: "vllm" | "llama.cpp" | "sglang" | "ds4" | "exl3" | "q27" | null; modelId: string | null; modelPath: string | null; contextLength: number | null; diff --git a/src/components/OverviewPage/OverviewPage.tsx b/src/components/OverviewPage/OverviewPage.tsx index c0fa3230..70ae7260 100644 --- a/src/components/OverviewPage/OverviewPage.tsx +++ b/src/components/OverviewPage/OverviewPage.tsx @@ -339,7 +339,9 @@ function SparkCard({ ? "sgLang" : llm.backend === "exl3" ? "EXL3" - : llm.backend ?? "LLM" + : llm.backend === "q27" + ? "q27" + : llm.backend ?? "LLM" } value={llm.modelId ?? "unknown"} tone="accent" diff --git a/src/components/SparkPage/LlmPanel.tsx b/src/components/SparkPage/LlmPanel.tsx index b265b34f..3c29ee11 100644 --- a/src/components/SparkPage/LlmPanel.tsx +++ b/src/components/SparkPage/LlmPanel.tsx @@ -24,15 +24,15 @@ const VLLM_METRIC_INFO = { requests: "Run = requests actively generating on the GPU. Wait = accepted but not yet scheduled (capacity or constraints). Growing wait with high KV cache usually means the server is overloaded.", ttftP95: - "95th percentile time-to-first-token from vLLM’s history of requests: how long “slow” requests wait until the first output token. Spikes mean queueing, long prefills, or cold paths—not average decode speed.", + "95th percentile time-to-first-token from the engine’s request history: how long “slow” requests wait until the first output token. Spikes mean queueing, long prefills, or cold paths—not average decode speed.", preempts: "Cumulative times the engine paused a running request to free KV cache for others. Rising under load signals memory pressure; zero is normal when the server is comfortable.", prefixCache: "Lifetime fraction of prefix-cache lookups that hit (hits ÷ queries). Higher means more prompt reuse and less prefill work; — when the series is missing or unused.", e2eP95: - "95th percentile end-to-end request latency from vLLM’s history: arrival until the request finishes. Includes queue wait, prefill, and decode—not just token generation speed.", + "95th percentile end-to-end request latency from the engine’s request history: arrival until the request finishes. Includes queue wait, prefill, and decode—not just token generation speed.", itlP95: - "95th percentile inter-token latency (time between successive output tokens) from vLLM’s history. Spikes mean decode stalls or contention; lower is smoother streaming.", + "95th percentile inter-token latency (time between successive output tokens) from the engine’s request history. Spikes mean decode stalls or contention; lower is smoother streaming.", mtpAccept: "Lifetime speculative / MTP acceptance rate (accepted draft tokens ÷ drafted tokens). Higher means speculative decoding is paying off; — when speculation is off or unused.", } as const; @@ -47,6 +47,7 @@ function BackendBadge({ backend }: { backend: string | null }) { sglang: "sgLang", ds4: "ds4", exl3: "EXL3", + q27: "q27", }; return ( @@ -577,7 +578,7 @@ export function LlmPanel({ - {llm?.backend === "vllm" && ( + {llm && (llm.backend === "vllm" || llm.backend === "q27") && (
- {llm.requestsRunning != null && llm.requestsWaiting != null - ? `${Math.round(llm.requestsRunning)} run / ${Math.round(llm.requestsWaiting)} wait` + {llm.requestsRunning != null + ? `${Math.round(llm.requestsRunning)} run${ + llm.requestsWaiting != null + ? ` / ${Math.round(llm.requestsWaiting)} wait` + : "" + }` : "—"}
@@ -648,7 +653,7 @@ export function LlmPanel({
)} - {llm?.backend === "vllm" && ( + {llm && (llm.backend === "vllm" || llm.backend === "q27") && (