Size each machine by what SLURM granted it on mixed clusters (#617)

[wshlavacek]

Fixes #617.

The problem

When PyBNF starts workers across several machines it sends the same worker count to every one. On a cluster whose machines differ in size, that single count is wrong for some of them. Too many workers on a small machine leaves them competing for its processors. Too few on a large machine leaves it partly idle. Nothing stops the run or prints anything wrong. The fit is just slower than the allocation could have made it.

The reporter's cluster has a single queue whose machines differ in processor count by more than a factor of two, with separate requests landing on unequal machines.

The fix

This applies only to the srun launcher (cluster_type = slurm-srun, added in #614). dask ssh takes one worker count for all hosts and cannot express a per-machine count, which is why the SSH launcher is left unchanged.

The srun launcher now works out each machine's worker count from what SLURM granted it ($SLURM_JOB_CPUS_PER_NODE) and starts that many there. A single srun job step binds every machine in it to the same number of CPUs, so a mixed allocation is started as one job step per distinct machine size, each on its own machines, which SLURM runs at the same time.

For example, a run on two 40-processor machines and one 96-processor machine starts 40 workers on each of the first two and 96 on the third. The per-machine arrangement is written to the log, and each job step writes its own worker log (dask_workers.log, dask_workers_2.log, and so on) so their output does not interleave.

An allocation whose machines are all the same size, which is the common case, still runs as one job step, exactly as before (the pre-change command is pinned byte-for-byte by a regression test). If SLURM does not report a per-machine list PyBNF can line up with its machines, it falls back to sizing every machine the same and warns that it did.

Readiness now waits for every worker the steps should produce, not just the first, so a job step whose placement failed is caught rather than masked by another that succeeded.

Setting parallel_count still splits that total evenly across the machines on either launcher. Making that override per-machine is deliberately out of scope, and a known limitation of it is noted in the ADR.

What is in the PR

• The fix in pybnf/cluster.py.
• ADR-0124 explaining the decision, the fallback, the stronger readiness check, and why parallel_count is left as an even split.
• A CHANGELOG entry and an update to docs/cluster.rst.
• Tests in tests/test_cluster.py: the per-machine count helpers, the grouped srun commands, the heterogeneous and homogeneous and parallel_count and fallback cases of the bring-up, the multi-process readiness check, and an extension of the installed-SLURM check for the grouped command's srun options.

Verification

python -m pytest tests/test_cluster.py -q passes (179 tests). The tests take the constructed command string as the oracle, matching this module's existing philosophy. That SLURM places the concurrent per-group steps as intended on a real heterogeneous allocation is what the reporter's cluster verifies, the same way #614 was confirmed.