Remote metric polling opens ~311,000 short-lived SSH logins per remote node per day, causing session churn and memory growth

[iwakin999]

Summary

In the documented/default Docker setup, remote monitoring invokes a new OpenSSH client for every sshExec() call. Because sparkDash does not configure connection multiplexing, each call normally establishes a new authenticated SSH connection and PAM session. With the default collector intervals on a healthy remote Spark, this produces a nominal steady-state rate of approximately 3.6 SSH authentications and PAM session opens per second (about 311,000 of each per day), even when no WebSocket clients are connected.

This creates avoidable sshd, PAM, systemd-logind, journal, and downstream session-monitor churn on every remote node. On one continuously monitored Ubuntu 24.04 DGX Spark, the rate calculated from the call sites matched the observed SSH authentication and PAM session-open counts during the measurement window.

The user-facing UI and deployment model do not need to change. The existing OpenSSH client could reuse one authenticated transport per effective remote connection via multiplexing, with the current one-shot behavior retained as a compatibility fallback.

Environment

• sparkDash main at 7b47cd11d09adeee9e293d6f221430314d9e1ac6
• Docker deployment on a head DGX Spark
• One remote DGX Spark monitored with key-based SSH authentication
• OpenSSH client 9.2p1 in the sparkDash container
• Remote node: Ubuntu 24.04 LTS, OpenSSH server with PAM/systemd-logind
• LLM, Hermes, and Tailnet monitoring disabled on the remote worker; the measurements below are from system-metric and liveness polling

The remote-node polling path is independent of browser access or any client-side SSH tunnel. These connections originate from the sparkDash container on the head node using its configured monitoring key.

Observed behavior

Over a 120-second window on the remote node:

Accepted publickey:                         434
pam_unix(sshd:session): session opened:    429

That is approximately 3.6 new authenticated sessions per second, matching the rate derived from the source.

The monitor continues polling in the background with no WebSocket clients, as documented in the README.

Correlated downstream observation

During the same workload, polkitd anonymous RSS increased while the live logind-session count returned to baseline. One snapshot after about 66 minutes showed:

VmRSS:      ~76 MiB
RssAnon:    ~69 MiB
RssFile:     ~7 MiB
live logind sessions: 2

This does not establish that sparkDash causes a polkitd memory leak. The observation is included only as a possible downstream effect of sustained session-lifecycle activity. The independently reproducible issue reported here is the repeated SSH authentication and PAM session creation.

Root cause

server/collectors/ssh.js starts a new OpenSSH client invocation for each sshExec() call via execFile() (ssh for key authentication, or sshpass wrapping ssh for password authentication):

sparkDash/server/collectors/ssh.js

Lines 65 to 148 in 7b47cd1

	export async function sshExec(spark, cmd, options = {}) {
	const timeoutMs =
	Number.isFinite(options.timeoutMs) && options.timeoutMs > 0 ? options.timeoutMs : 10000;
	const { host, user, auth, password } = spark.ssh || {};
	const targetHost = host || spark.lanIp;
	if (!targetHost || !user) {
	throw new Error(`SSH config missing for ${spark.id}: host=${targetHost}, user=${user}`);
	}
	if (!isAllowedTargetHost(targetHost)) {
	throw new Error(`SSH host not allowed: ${targetHost}`);
	}
	if (!isValidSshUser(user)) {
	throw new Error(`SSH user not allowed: ${user}`);
	}
	if (typeof cmd !== "string" || !cmd) {
	throw new Error("SSH command must be a non-empty string");
	}
	// Base SSH options (no shell metacharacters in argv)
	// accept-new: trust first-seen host key (LAN ops); pin known_hosts for stricter envs
	const baseOpts = [
	"-o",
	`ConnectTimeout=${SSH_CONNECT_TIMEOUT}`,
	"-o",
	"StrictHostKeyChecking=accept-new",
	];
	const remote = `${user}@${targetHost}`;
	// Remote command as a single argument — ssh does not invoke a local shell for it
	// when using execFile without a shell. `--` stops option parsing before destination.
	let file;
	let args;
	// Minimal child env — only what ssh/sshpass actually need. Spreading the full
	// `process.env` would leak every host var (AWS_*, GITHUB_TOKEN, etc.) into the
	// child; this whitelist scopes to PATH, HOME, USER/LOGNAME (ssh logging +
	// known_hosts mixing), TERM, and SSH_AUTH_SOCK so agent-forwarded key auth
	// still works. SSHPASS is added below only for password auth.
	const env = {
	PATH: process.env.PATH || "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
	HOME: process.env.HOME || "/root",
	USER: process.env.USER,
	LOGNAME: process.env.LOGNAME,
	TERM: process.env.TERM || "xterm",
	...(process.env.SSH_AUTH_SOCK ? { SSH_AUTH_SOCK: process.env.SSH_AUTH_SOCK } : {}),
	};
	if (auth === "pass") {
	if (!password) {
	throw new Error(
	`SSH password auth selected for ${spark.id} but no password is set (Edit Spark once — passwords are stored encrypted and survive restarts)`
	);
	}
	if (!sshpassAvailable()) {
	throw new Error(`sshpass is not installed. Install it with: sudo apt-get install sshpass`);
	}
	// Password via env (sshpass -e) — never on argv or in process list as -p
	env.SSHPASS = password;
	file = "sshpass";
	args = ["-e", "ssh", ...baseOpts, "--", remote, cmd];
	} else {
	// Key-based SSH (default) — BatchMode prevents hanging on missing keys
	file = "ssh";
	args = [...baseOpts, "-o", "BatchMode=yes"];
	const identityFile = process.env.SSH_IDENTITY_FILE;
	if (identityFile) {
	args.push("-i", identityFile);
	}
	args.push("--", remote, cmd);
	}
	return new Promise((resolve, reject) => {
	execFile(file, args, { timeout: timeoutMs, env, maxBuffer: 10 * 1024 * 1024 }, (err, stdout, stderr) => {
	if (err) {
	const msg = stderr?.trim() || err.message;
	reject(new Error(`SSH to ${targetHost} failed: ${msg}`));
	} else {
	resolve(String(stdout).trim());
	}
	});
	});
	}

There is no ControlMaster, ControlPersist, connection pool, or other persistent SSH transport.

SparkMonitor.start() schedules metric domains independently:

sparkDash/server/sparks/SparkMonitor.js

Lines 341 to 359 in 7b47cd1

	/** Start background polling. */
	start() {
	if (this._running) return;
	this._running = true;
	this._stopped = false;
	this._poll();
	this._intervals.push(setInterval(() => this._pollDomain("gpu"), POLL_INTERVAL_GPU));
	this._intervals.push(setInterval(() => this._pollDomain("cpu"), POLL_INTERVAL_CPU));
	this._intervals.push(setInterval(() => this._pollDomain("network"), POLL_INTERVAL_NETWORK));
	this._intervals.push(setInterval(() => this._pollDomain("storage"), POLL_INTERVAL_STORAGE));
	this._intervals.push(setInterval(() => this._pollDomain("ram"), POLL_INTERVAL_CPU));
	this._intervals.push(setInterval(() => this._pollDomain("memory"), POLL_INTERVAL_BANDWIDTH));
	this._restartLlmPollInterval();
	this._restartComfyPollInterval();
	this._restartHermesPollInterval();
	this._restartTailscalePollInterval();
	// Liveness on a slightly slower cadence
	this._intervals.push(setInterval(() => this._checkOnline(), POLL_INTERVAL_LIVENESS));
	console.log(`[SparkMonitor] ${this.spark.id} started`);

For a healthy target, with storage auto-polling enabled, a primary network interface detected, and the optional Hermes and Tailnet SSH probes disabled, the nominal steady-state rate is:

Domain	Default interval	New authenticated SSH connections
GPU	2 s	0.5/s
CPU	2 s	0.5/s
RAM	2 s	0.5/s
Network	2 s, normally two sshExec() calls; link-speed read is conditional	1.0/s
Unified memory	2 s	0.5/s
Storage	5 s	0.2/s
Liveness + uptime	5 s, two sshExec() calls	0.4/s
Total		~3.6/s

Remote collector call sites:

sparkDash/server/collectors/SystemCollector.js

Lines 921 to 1272 in 7b47cd1

	async _getRemoteGpu() {
	try {
	const cmd = [
	"nvidia-smi --query-gpu=temperature.gpu,utilization.gpu,power.draw,power.limit,clocks.current.sm,clocks.max.sm,clocks_throttle_reasons.hw_thermal_slowdown,clocks_throttle_reasons.sw_thermal_slowdown,clocks_throttle_reasons.hw_slowdown,clocks_throttle_reasons.sw_power_cap --format=csv,noheader,nounits 2>/dev/null",
	"echo '---'",
	"nvidia-smi --query-gpu=memory.used,memory.total --format=csv,noheader,nounits 2>/dev/null",
	"echo '---'",
	"nvidia-smi --query-compute-apps=pid,process_name,used_gpu_memory --format=csv,noheader,nounits 2>/dev/null",
	"echo '---'",
	"grep -E 'MemTotal|MemAvailable' /proc/meminfo 2>/dev/null",
	].join("; ");
	const output = await sshExec(this.spark, cmd);
	const sections = output.split("---");
	const gpuOut = sections[0]?.trim() || "";
	const memFields = sections[1]?.trim() || "";
	const computeOut = sections[2]?.trim() || "";
	const meminfoOut = sections[3]?.trim() || "";
	const gpu = this._parseGpuLine(gpuOut);
	// Parse memory.used / memory.total from nvidia-smi (may be [N/A] on GB10)
	let used = null;
	let total = null;
	const memLine = memFields.split("\n").filter(Boolean)[0] || "";
	const memParts = memLine.split(",").map((s) => s.trim());
	used = this._parseSmiNumber(memParts[0]);
	total = this._parseSmiNumber(memParts[1]);
	const apps = this._parseComputeApps(computeOut);
	this.nvidiaComputeAppsCache.clear();
	let computeSum = 0;
	for (const app of apps) {
	this.nvidiaComputeAppsCache.set(app.pid, { name: app.name, vramMB: app.vramMB });
	computeSum += app.vramMB;
	}
	if ((used == null || used === 0) && computeSum > 0) used = computeSum;
	// Unified-memory pool: prefer MemTotal (OS-visible) so VRAM and Unified
	// Memory panels share the same base. Available = MemAvailable (real free).
	const totalMatch = meminfoOut.match(/MemTotal:\s+(\d+)\s+kB/);
	const availMatch = meminfoOut.match(/MemAvailable:\s+(\d+)\s+kB/);
	const memTotalMB = totalMatch ? Math.round(parseInt(totalMatch[1]) / 1024) : 0;
	let availableMB = availMatch ? Math.round(parseInt(availMatch[1]) / 1024) : 0;
	const usedMB = Math.round(used || 0);
	let totalMB = Math.round(total || 0);
	if (this.spark.kind === "host") {
	// Discrete GPU VRAM: trust nvidia-smi's memory.total; free VRAM = total − used.
	if (totalMB <= 0 && memTotalMB > 0) totalMB = memTotalMB;
	else if (totalMB <= 0) totalMB = DGX_SPARK.MEMORY_HBM_SIZE_GB * 1024; // Convert to MB
	if (totalMB > 0 && usedMB > 0) availableMB = Math.max(0, totalMB - usedMB);
	} else {
	// GB10 shared HBM pool: prefer the OS-visible pool (MemTotal) as the total,
	// fall back to nvidia-smi, then the hardware spec (HBM) only if nothing known.
	if (memTotalMB > 0) totalMB = memTotalMB;
	else if (totalMB <= 0) totalMB = DGX_SPARK.MEMORY_HBM_SIZE_GB * 1024; // Convert to MB
	}
	const percentage = totalMB > 0 ? Math.round((usedMB / totalMB) * 100) : 0;
	// Rough system power estimate: GPU draw + 20W CX7/peripherals
	const systemDraw = Math.round(gpu.powerDraw + 20);
	// Top 5 GPU processes by VRAM usage
	const processes = Array.from(this.nvidiaComputeAppsCache.entries())
	.map(([pid, info]) => ({ pid, name: info.name, vramMB: info.vramMB }))
	.sort((a, b) => b.vramMB - a.vramMB)
	.slice(0, 5);
	return {
	temperature: gpu.temperature,
	usage: gpu.usage,
	power: { draw: gpu.powerDraw, limit: gpu.powerLimit, systemDraw },
	vram: { used: usedMB, total: totalMB, percentage, available: availableMB },
	processes,
	throttle: gpu.throttle,
	};
	} catch (err) {
	console.error(`[SystemCollector] Remote GPU error for ${this.spark.id}:`, err.message);
	return this._defaultGpu();
	}
	}
	/**
	* One SSH round trip: /proc/stat, CPU arch, then the same hwmon-then-thermal
	* sensor dump local `_getCPUTemperature()` uses. `|| true` on the thermal
	* glob keeps a missing zone from failing the whole CPU poll (sshExec treats
	* any non-zero exit as a hard error).
	*/
	_buildRemoteCpuCommand() {
	return [
	"cat /proc/stat | head -1",
	"echo '---'",
	"cat /proc/cpuinfo | grep -E 'CPU architecture|aarch64' | head -1",
	"echo '---'",
	// GB10 also exposes nvme/mlx5 sensors; the name allowlist keeps those out.
	'for h in /sys/class/hwmon/*; do n=$(cat "$h/name" 2>/dev/null); case "$n" in coretemp|k10temp|zenpower|acpitz) for t in "$h"/temp*_input; do cat "$t" 2>/dev/null; break; done;; esac; done',
	"cat /sys/class/thermal/thermal_zone*/temp 2>/dev/null || true",
	].join("; ");
	}
	async _getRemoteCpu(sshExecutor = sshExec) {
	try {
	const cmd = this._buildRemoteCpuCommand();
	const output = await sshExecutor(this.spark, cmd);
	const sections = output.split("---");
	const statOut = sections[0]?.trim() || "";
	const cpuinfoOut = sections[1]?.trim() || "";
	const tempOut = sections[2] || "";
	const cpuStat = this._parseCPUUsage(statOut);
	const totalDiff = cpuStat.total - (this.lastCpuStat?.total || cpuStat.total);
	const usedDiff = cpuStat.used - (this.lastCpuStat?.used || cpuStat.used);
	const usage = totalDiff > 0 ? Math.round((usedDiff / totalDiff) * 100) : 0;
	this.lastCpuStat = cpuStat;
	// ARM/Neoverse power estimation
	const isArm = /CPU architecture:\s*[89]|aarch64|ARMv[89]|armv[89]/i.test(cpuinfoOut);
	const tdp = isArm ? 65 : 185;
	const idleWatts = tdp * 0.08;
	const draw = idleWatts + (tdp - idleWatts) * Math.min(usage / 100, 1);
	return {
	usage,
	temperature: this._parseSensorTemp(tempOut),
	draw: Math.round(draw * 10) / 10,
	tdp: Math.round(tdp),
	};
	} catch (err) {
	console.error(`[SystemCollector] Remote CPU error for ${this.spark.id}:`, err.message);
	return this._defaultCpu();
	}
	}
	/**
	* First plausible temperature from a remote sensor dump (raw millidegrees,
	* one per line, highest priority first). Same accept range as local
	* `_getCPUTemperature()`; returns 0 when nothing is readable.
	*
	* @param {string} raw
	* @returns {number} degrees Celsius, or 0
	*/
	_parseSensorTemp(raw) {
	for (const line of String(raw).split("\n")) {
	const millidegrees = parseInt(line.trim(), 10);
	if (Number.isFinite(millidegrees) && millidegrees > 0 && millidegrees < 200000) {
	return Math.round((millidegrees / 1000) * 10) / 10;
	}
	}
	return 0;
	}
	async _getRemoteRam() {
	try {
	const cmd = "grep -E 'MemTotal|MemAvailable' /proc/meminfo 2>/dev/null";
	const output = await sshExec(this.spark, cmd);
	const totalMatch = output.match(/MemTotal:\s+(\d+)\s+kB/);
	const availMatch = output.match(/MemAvailable:\s+(\d+)\s+kB/);
	const totalKB = totalMatch ? parseInt(totalMatch[1]) : 0;
	const availKB = availMatch ? parseInt(availMatch[1]) : 0;
	const usedKB = totalKB - availKB;
	return {
	used: Math.round(usedKB / 1024),
	total: Math.round(totalKB / 1024),
	percentage: totalKB > 0 ? Math.round((usedKB / totalKB) * 100) : 0,
	};
	} catch (err) {
	console.error(`[SystemCollector] Remote RAM error for ${this.spark.id}:`, err.message);
	return this._defaultRam();
	}
	}
	async _getRemoteStorage() {
	try {
	// Include root (/); exclude pseudo filesystems via -x and type filter
	const cmd =
	"df -l -B1 -T -x tmpfs -x devtmpfs -x squashfs -x overlay -x efivarfs -x proc -x sysfs -x devpts -x cgroup -x cgroup2 2>/dev/null";
	const output = await sshExec(this.spark, cmd);
	const lines = output.trim().split("\n").slice(1); // Skip header
	const disks = [];
	const disabledDevices = this.spark.disabledDevices || [];
	const PSEUDO = new Set([
	"tmpfs",
	"devtmpfs",
	"proc",
	"sysfs",
	"efivarfs",
	"squashfs",
	"overlay",
	"devpts",
	"cgroup",
	"cgroup2",
	]);
	for (const line of lines) {
	const parts = line.split(/\s+/);
	if (parts.length < 7) continue;
	const [fsys, type, size, used, avail, pct, mount] = parts;
	if (mount === "/boot/efi" || mount.includes("/snap")) continue;
	if (PSEUDO.has((type || "").toLowerCase())) continue;
	const device = fsys.split("/").pop() || fsys;
	const isDisabled =
	disabledDevices.includes(device) || disabledDevices.includes(mount);
	disks.push({
	device,
	label: mount,
	used: Math.round(parseInt(used) / 1024 / 1024),
	total: Math.round(parseInt(size) / 1024 / 1024),
	available: Math.round(parseInt(avail) / 1024 / 1024),
	percentage: parseInt(pct) || 0,
	readSpeed: 0,
	writeSpeed: 0,
	disabled: isDisabled,
	});
	}
	return disks;
	} catch (err) {
	console.error(`[SystemCollector] Remote Storage error for ${this.spark.id}:`, err.message);
	return [];
	}
	}
	async _getRemoteNetwork() {
	try {
	const cmd = [
	"cat /proc/net/dev 2>/dev/null",
	"echo '---'",
	"cat /proc/net/route 2>/dev/null",
	"echo '---'",
	"ip -4 addr show 2>/dev/null",
	"echo '---'",
	// Collect operstate for all non-virtual interfaces in one go
	"for d in /sys/class/net/*/operstate; do echo \"$(basename $(dirname $d)):$(cat $d)\"; done",
	"echo '---'",
	// WoL MAC for the primary LAN NIC on DGX Spark
	`cat /sys/class/net/${WOL_INTERFACE}/address 2>/dev/null || true`,
	].join("; ");
	const output = await sshExec(this.spark, cmd);
	const sections = output.split("---");
	const devOut = sections[0]?.trim() || "";
	const routeOut = sections[1]?.trim() || "";
	const ipOut = sections[2]?.trim() || "";
	const operstateOut = sections[3]?.trim() || "";
	const wolMac = normalizeMac(sections[4]?.trim() || "");
	// Parse operstate lines ("enP7s7:up")
	const operstateMap = new Map();
	for (const line of operstateOut.split("\n")) {
	const idx = line.indexOf(":");
	if (idx > 0) {
	operstateMap.set(line.slice(0, idx), line.slice(idx + 1).trim().toLowerCase());
	}
	}
	// Parse IP addresses
	const ipMap = new Map();
	const ipBlocks = ipOut.split(/\n(?=\d+:\s+)/);
	for (const block of ipBlocks) {
	const first = block.split("\n")[0];
	const m = first.match(/^\d+:\s+(\S+):/);
	if (!m) continue;
	const iface = m[1];
	const ipMatch = block.match(/inet\s+([\d.]+)/);
	if (ipMatch) {
	ipMap.set(iface, ipMatch[1]);
	}
	}
	// Parse /proc/net/dev
	const lines = devOut.split("\n").slice(2);
	const now = Date.now();
	const interfaces = [];
	for (const line of lines) {
	const parts = line.trim().split(/[\s:]+/);
	if (parts.length < 17) continue;
	const iface = parts[0];
	if (this._isVirtualNetworkInterface(iface)) continue;
	const rxBytes = parseInt(parts[1]) || 0;
	const txBytes = parseInt(parts[9]) || 0;
	const last = this.lastNetworkStats.get(iface) || { rxBytes, txBytes, time: now };
	const dtSec = (now - last.time) / 1000;
	const rxSpeed = dtSec > 0 ? (rxBytes - last.rxBytes) / dtSec : 0;
	const txSpeed = dtSec > 0 ? (txBytes - last.txBytes) / dtSec : 0;
	this.lastNetworkStats.set(iface, { rxBytes, txBytes, time: now });
	interfaces.push({
	name: iface,
	rxSpeed: Math.max(0, Math.round(rxSpeed)),
	txSpeed: Math.max(0, Math.round(txSpeed)),
	ip: ipMap.get(iface) || null,
	operstate: operstateMap.get(iface) || "unknown",
	disabled: false,
	});
	}
	const tagged = this._tagDisabledInterfaces(interfaces);
	// Parse /proc/net/route for default interface
	let primaryInterface = null;
	const routeLines = routeOut.split("\n");
	for (const line of routeLines) {
	const parts = line.trim().split(/\s+/);
	if (parts.length >= 11 && parts[1] === "00000000" && (parseInt(parts[3], 16) & 1)) {
	primaryInterface = parts[0];
	break;
	}
	}
	if (primaryInterface && (this.spark.disabledInterfaces || []).includes(primaryInterface)) {
	const alt = tagged.find((i) => !i.disabled);
	primaryInterface = alt?.name ?? primaryInterface;
	}
	let linkSpeedMbps = null;
	if (primaryInterface) {
	try {
	// Interface name is from the kernel; still keep it to safe chars
	if (/^[a-zA-Z0-9._-]+$/.test(primaryInterface)) {
	const speedRaw = await sshExec(
	this.spark,
	`cat /sys/class/net/${primaryInterface}/speed 2>/dev/null || true`
	);
	const n = parseInt(String(speedRaw).trim(), 10);
	if (Number.isFinite(n) && n > 0) linkSpeedMbps = n;
	}
	} catch {
	/* link speed optional */
	}
	}
	return { primaryInterface, linkSpeedMbps, interfaces: tagged, wolMac };
	} catch (err) {
	console.error(`[SystemCollector] Remote Network error for ${this.spark.id}:`, err.message);
	return this._defaultNetwork();
	}
	}
	async _getRemoteUnifiedMemory() {
	try {
	const cmd = [
	"grep -E 'MemTotal|MemAvailable' /proc/meminfo 2>/dev/null",
	"echo '---'",
	"nvidia-smi --query-compute-apps=pid,process_name,used_gpu_memory --format=csv,noheader,nounits 2>/dev/null",
	].join("; ");
	const output = await sshExec(this.spark, cmd);

The Settings pollIntervalMs value is only the WebSocket snapshot broadcast interval; changing it does not change these collector intervals.

Reproduction

1. Run sparkDash on a head node.
2. Add one remote Spark using key-based SSH authentication.
3. Leave the sparkDash server/container running with default metric intervals. A browser does not need to remain connected.
4. Use an otherwise idle remote node, or locally filter journal entries to the sparkDash source/user before counting. Do not include private addresses or usernames in public output.
5. Capture one fixed 120-second window, then count SSH authentications and PAM sessions over that same window:

start="$(date -Is)"
sleep 120
end="$(date -Is)"

sudo journalctl -u ssh --since "$start" --until "$end" --no-pager -o cat \
  | grep -c "Accepted publickey"

sudo journalctl -u ssh --since "$start" --until "$end" --no-pager -o cat \
  | grep -c "pam_unix(sshd:session): session opened"

On distributions using sshd.service, replace -u ssh with -u sshd.

6. Stop sparkDash and repeat the same fixed-window measurement. The authentication/session counts attributable to sparkDash should fall to baseline in the next measurement window.

Expected behavior

• One authenticated SSH transport should be reused per remote Spark.
• The metric refresh cadence should remain unchanged.
• New SSH authentication/PAM sessions should occur only when the connection is first established or re-established after failure.
• Users should not need to install an agent, open another port, add another key, change sshd_config, or modify their Compose deployment.

Proposed direction

Keep the existing sshExec(spark, cmd, options) API and make connection reuse an internal implementation detail:

1. Maintain one OpenSSH master per effective destination and security context, including host, port, user, authentication/credential identity, and connection-defining SSH options. Never include raw passwords or other secrets in the control path or logs.
2. Use ControlMaster, ControlPersist, and a hashed ControlPath under a private (0700) runtime directory inside the container.
3. Serialize initial master creation so simultaneous startup polls cannot race.
4. Check/reconnect a failed master automatically with bounded backoff.
5. Close the correct master when a Spark is removed or its SSH settings change, and clean up all masters on server shutdown.
6. Detect and safely handle stale control sockets.
7. Preserve the current host-key verification, non-interactive authentication, command-argument handling, connection timeout, and per-call timeout semantics.
8. Fall back safely to the current one-shot SSH behavior if multiplexing cannot be established or a new session channel is rejected, preserving compatibility and emitting a rate-limited warning so sustained fallback mode is observable.

This requires no additional runtime package: the production image already installs openssh-client, and the supported OpenSSH client implements these options.

As a complementary optimization, the frequent system domains could later be combined into one remote collection command and liveness inferred from collection success. That would reduce channel/process overhead further, but it is not required to eliminate repeated authentication/PAM sessions.

Acceptance criteria

• With one healthy remote Spark and default intervals, Accepted publickey and PAM session-open events fall from ~3.6/s to approximately one at initial connection and one per actual reconnect.
• GPU, CPU, RAM, network, storage, unified-memory, uptime, and online-state results remain equivalent to the current implementation.
• Monitoring recovers after killing the master connection or rebooting the remote node.
• Updating/removing a Spark does not reuse the old host/user/key connection.
• Connections that differ by host, port, user, credential identity, or connection-defining SSH options never share a master.
• Existing host-key verification and timeout behavior remains unchanged.
• Stopping/restarting the sparkDash container leaves no external control socket or SSH master behind.
• Key authentication remains non-interactive; password authentication retains its current behavior or explicitly falls back to one-shot SSH.
• Tests cover concurrent first use, stale sockets, reconnect, configuration changes, and fallback.

Workarounds

Changing the Settings pollIntervalMs value does not reduce collector SSH activity because it controls only WebSocket snapshot delivery. Stopping sparkDash stops the observed churn; no confirmed workaround currently preserves remote monitoring while avoiding repeated authentication.