docs(cluster): commands that exist, a plain answer on SSH logins, and how to size a fit to the processors reserved (#622)

[wshlavacek]

Closes #622.

docs/cluster.rst is the only guidance for running PyBNF on a cluster, and parts of it described a program that no longer exists, answered a failure with advice that cannot fix it, and said nothing about the setting that decides whether an allocation gets used. This is a prose-only change: no key changes meaning, no code is touched, and sphinx-build -W --keep-going is clean.

The command names

Already corrected on main by #631, which renamed them in the code and the docs together. Verified here that no dask-scheduler / dask-worker / dask-ssh spelling survives outside the CHANGELOG, the ADR, and the tests that pin the old names on purpose.

Which ways of starting a run log in to other machines

The old "Troubleshooting: SSH access to nodes" treated every multi-machine failure as a missing key. On the cluster in #614 that was not the obstacle: logging in by hand worked and the run still failed, because dask ssh does not run ssh — it logs in through paramiko, which offers a public key or a typed password and nothing else.

The section now opens with a table of which ways involve a login at all:

How the run is started	Logs in?
-t slurm	Yes — dask ssh to every allocated node
scheduler_node / worker_nodes	Yes — same launcher, node names you supplied
-t slurm-srun	No
-s cluster.json	No
single machine	No other machines

and then hands the reader the login PyBNF actually attempts, as a one-line paramiko connection mirroring distributed.deploy.old_ssh argument for argument, with its three outcomes separated: it succeeds; it raises AuthenticationException where plain ssh succeeds (no key will help — use -t slurm-srun); or it asks for a password, which is the one case ssh-keygen fixes.

Sizing a run

New section connecting the processors reserved to the number PyBNF can keep busy: the workers (what the job was granted) against the simulations in flight (population_size for almost every algorithm), with a table of what each fit_type can really run at once — population_size for the population methods and samplers, population_size × (population_size − 1) for scatter search, min(population_size, N−1) per start for Simplex, one per start for Powell, two walks per parameter for profile likelihood, and none for hmc, whose chains never reach a worker. Includes the wave arithmetic for synchronized algorithms (129 parameter sets on 128 workers costs what 256 would) and a walk-through of the shipped tcr example, whose population_size = 9 is exactly the 72 processors its batch script reserves.

Where simulation files are written

The simulation_dir advice read as general when it is not. The new section says the default is right on the ordinary network filesystem most clusters give you, that the key exists for a parallel filesystem such as Lustre or GPFS, and adds the requirement worth writing down: the directory must be visible from the node PyBNF runs on, since that is where the best fit's output is copied from when delete_old_files = 0.

Example batch scripts

The nine scripts telling the reader to load Anaconda for "Python 3.5" now carry a marked line to edit for a Python 3.11+ environment. examples/tcr/tcr_batch.sh and examples/tlbr/tlbr_batch.sh also reserve processors matching their own configs (4 × 18 = 72 for a reference set of 9; 5 × 30 = 150 for 250 parameter sets × 3 smoothing replicates), so the examples the page points at demonstrate the sizing section rather than contradicting it.

Also corrected in passing

docs/algorithms.rst claimed Simplex fans out to parallel_count processors and could use N+1 of them. It uses min(population_size, N−1) per start, times n_starts concurrent starts, and has since #498.