fix(cluster): a failed worker start-up says what went wrong and what to try instead (#618)

[wshlavacek]

Closes #618.

The problem

A multi-machine run whose workers could not be started stopped with

Failed to start the dask-ssh cluster (dask-ssh exited with code 1)

and, on the cluster this was reported from, nothing else. The real cause was that the login to the other machines had failed. Nothing in the message said so, named a cause, or named a way of running that needs no login — and since the failure ends the run, that message is the entirety of what the user gets.

Stopping is right, and the issue says so: carrying on with fewer machines than were asked for would waste the whole run, and the failure is reported in about ten seconds. What was missing is why.

Why the message was empty

The existing code did include the captured output "when there is any". There was none, and that is not a coincidence — the half of dask's output that explains the failure was discarded twice over:

• dask ssh explains itself on stdout. When a login fails it prints the node it was connecting to and the exception paramiko raised, and lets only the traceback fall to stderr. PyBNF captured stderr and sent stdout to DEVNULL.
• dask exits without flushing. A failed bring-up ends in os._exit(1), which does not flush Python's buffers. stdout writing to a file is block-buffered, and dask's few hundred bytes never reach the 8 KB that would force a write, so they are dropped at exit.

Measured against dask 2026.7.1 on a login that fails: 0 of dask's own lines survived; 15 survive now that both streams are captured into one file and dask is run with PYTHONUNBUFFERED=1.

What the message says now

• It quotes what dask said, and says plainly when there was nothing to quote rather than falling back to "Check the cluster log directory for details" without naming a directory.
• The Python traceback frames are folded out of what it quotes — a failed login writes one traceback per node per retry, three retries each, and the sentences that say what happened are buried in dask's and paramiko's own source. 137 captured lines became 32, losing none of those sentences. The log still keeps every line.
• When the output reads as a refused credential ("Authentication failed", "No authentication methods available", an encrypted key, a host key that did not match), it says the login is the likely cause and says what PyBNF logs in with: paramiko, which can offer a public key or a typed password and nothing else. That is what makes the failure survive ssh-keygen, and makes ssh othernode hostname succeeding from the same shell no evidence at all. A machine that could not be reached at all is deliberately not answered that way.
• Whatever the cause, it names both ways of running on several machines that need no login: cluster_type = slurm-srun (Automatic multi-machine startup cannot log in on clusters that use host-based or Kerberos SSH #614), which starts the workers inside the allocation SLURM already granted, and a scheduler_file naming a cluster that is already up.

The same failure, before and after — this is a real bring-up against a host paramiko cannot log in to (an encrypted ed25519 key, so it raises PasswordRequiredException):

Before — 40 lines of paramiko traceback, and nothing else:

Failed to start the dask ssh cluster (dask ssh exited with code 1). Details:
  File ".../paramiko/client.py", line 779, in _auth
    raise saved_exception
  ...
paramiko.ssh_exception.PasswordRequiredException: Private key file is encrypted

After:

Error: Could not start the workers on the other machines: dask ssh exited with code 1 during cluster bring-up.
This is what it said:
[ dask ssh ] : SSH connection error when connecting to 127.0.0.1:22 to run 'mkdir -p ...'
               SSH reported this exception: Private key file is encrypted
paramiko.ssh_exception.PasswordRequiredException: Private key file is encrypted
               Retrying... (attempt 1/3)
...
[ dask ssh ] : SSH connection failed after 3 retries. Exiting.
  -> This looks like a failed login. PyBNF starts the workers with `dask ssh`, which does not run your `ssh` command: it logs in with the paramiko library, which can offer a public key or a typed password and nothing else. A cluster whose nodes authenticate to each other by host-based or Kerberos (GSSAPI) SSH refuses that login however you configure it, and creating SSH keys does not help -- the cluster is not asking for a key. `ssh OTHERNODE hostname` succeeding proves nothing here; the "Running on a cluster" documentation gives a one-line test of the login PyBNF makes.
  -> Two ways of running on several machines need no login at all. On a SLURM cluster, cluster_type = slurm-srun (or pybnf -t slurm-srun) starts the workers with srun, inside the allocation SLURM already granted.
  -> The other: start a dask scheduler and workers yourself, by whatever means your cluster supports, and give PyBNF the scheduler file with -s or the scheduler_file key. PyBNF then only connects to a cluster that is already up.

Docs

docs/cluster.rst gains a paragraph on what a failed login looks like, in the section that already explains which ways of starting a run log in anywhere. docs/troubleshooting.rst gains an entry filed under the first line of the message, so the text a user sees is searchable.

Not in scope

The case where the program PyBNF runs does not exist at all is #615, already fixed: it is refused before anything is launched.

Verification

• A real failed bring-up on a developer machine, quoted above.
• Thirteen new assertions in tests/test_cluster.py, all red against the old code: that stdout is the stream captured, that PYTHONUNBUFFERED is set and the rest of the environment passed through, that traceback frames leave the message but stay in the log, that a login failure is named as one, that a network failure is not, that both no-login alternatives are always named, that silence is reported as silence, and that colour escapes are stripped.
• Full suite green (4492 passed, 25 skipped). Docs build clean under -W --keep-going.