SystemCollector CPU probe exits non-zero when thermal_zone glob doesn't match — 'Remote CPU error' for host-kind nodes

[ayayalar]

Summary

The CPU collector on kind: "host" nodes (SystemCollector.js) constructs a remote probe command whose final segment can exit non-zero, causing the entire SSH call to be treated as failed. CPU / temperature / power tiles stay broken (showing errors) for any host that lacks a matching /sys/class/thermal/thermal_zone* path, even though the CPU data itself is successfully fetched.

Environment

• sparkDash 1.8.2 (also present in 1.8.1)
• Affects kind: "host" nodes (e.g. x86 workstations/powerspecs). Does NOT affect kind: "spark" nodes (seen: sparks monitor fine).

Symptoms

• Recurring log spam, several times per minute on each affected node:
  [SystemCollector] Remote CPU error for <id>: SSH to <host> failed: Command failed: sshpass -e ssh ... -- <user>@<host> cat /proc/stat | head -1; ...
• Dashboard shows the node tile but CPU usage / temperature / draw never populate (falls back to _defaultCpu()).

Root cause

server/collectors/SystemCollector.js builds the remote command as:

const cmd = [
  "cat /proc/stat | head -1",
  "echo '---'",
  "cat /proc/cpuinfo | grep -E 'CPU architecture|aarch64' | head -1",
  ...(this.spark.kind === "host"
    ? [
        "echo '---'",
        'for h in /sys/class/hwmon/*; do n=$(cat "$h/name" 2>/dev/null); case "$n" in coretemp|k10temp|zenpower|acpitz) for t in "$h"/temp*_input; do cat "$t" 2>/dev/null; break; done;; esac; done',
        "cat /sys/class/thermal/thermal_zone*/temp 2>/dev/null",
      ]
    : []),
].join("; ");

On hosts where /sys/class/thermal/thermal_zone* does not exist (e.g. x86 systems whose temps come from hwmon/coretemp instead), the final cat /sys/class/thermal/thermal_zone*/temp 2>/dev/null gets an unexpanded glob, cat exits 1 (stderr suppressed), and because the segments are joined with ;, that 1 becomes the exit status of the whole remote shell command.

server/collectors/ssh.js (sshExec) treats any non-zero child exit as failure:

if (err) {
  const msg = stderr?.trim() || err.message;
  reject(new Error(`SSH to ${targetHost} failed: ${msg}`));
}

ssh exits with the remote command's status → sshpass reports Command failed: <argv> → the collector catches and logs Remote CPU error and returns _defaultCpu(). The stdout (real CPU stat + temperature from hwmon) is captured but discarded.

Why sparks are unaffected: the temperature probe segment is only appended for kind === "host"; for kind === "spark" the last segment is cat /proc/cpuinfo | grep -E 'CPU architecture|aarch64' | head -1, which exits 0 on GB10 (aarch64 matches).

Reproduction

From inside the container (or any host), with valid credentials:

sshpass -e ssh -o ConnectTimeout=5 -o StrictHostKeyChecking=accept-new -- \
  <user>@<x86-host> \
  "cat /proc/stat | head -1; echo ---; cat /proc/cpuinfo | grep -E 'CPU architecture|aarch64' | head -1; echo ---; for h in /sys/class/hwmon/*; do n=\$(cat \"\$h/name\" 2>/dev/null); case \"\$n\" in coretemp|k10temp|zenpower|acpitz) for t in \"\$h\"/temp*_input; do cat \"\$t\" 2>/dev/null; break; done;; esac; done; cat /sys/class/thermal/thermal_zone*/temp 2>/dev/null"
echo $?    # -> 1

Appending ; true makes the identical command exit 0 while returning all the data. Confirmed: same probe on the x86 host returned the full CPU stat, cpuinfo, and a coretemp value (~40 °C) — the data is present, only the trailing exit code breaks it.

Proposed fix

Make the probe's exit status not depend on a best-effort sensor read. Minimal change in SystemCollector.js (applies to all kinds, harmless):

].join("; ") + "; true";

Alternatively (equivalent, arguably clearer):

'cat /sys/class/thermal/thermal_zone*/temp 2>/dev/null || true',

Suggested hardening: audit the other sshExec probe commands assembled with ; for the same failure mode (any segment that can exit non-zero on a legitimate host — sensor reads, greps with no match, globs that might not expand). Long-term, consider having sshExec not reject when a non-empty stdout was produced (poll-style probes return data even when a tail command fails), or explicitly document that collectors must end probe commands with ; true.

Workaround for current builds

Patch SystemCollector.js as above and rebuild; no node/credential changes required.

Verified on sparkDash a3d6929 (local, on top of 1.8.2): after the fix, Remote CPU error spam stops and host CPU/temperature/draw populate on both affected powerspec nodes.

[vincenzopalazzo]

Fixed on main via _buildRemoteCpuCommand() ending with || true on the thermal_zone read (CPU temperature work). Related PR #68 is obsolete against current main — please close both when convenient.

— Vincent