test(cluster): check the cluster commands against the programs actually installed (#619)

[wshlavacek]

Closes #619.

The problem

The tests for starting a cluster checked that PyBNF built a particular command, comparing it against a copy of that command written into the test file. Nothing checked that the command could be run.

When distributed stopped installing dask-ssh (#615), every one of those tests kept passing while every real multi-machine run died on FileNotFoundError before a single simulation. Worse, the outdated name sat in the test file as the expected answer, so correcting PyBNF would have read as a test failure rather than as the fix.

Some of #619 was already addressed by #631, which introduced cluster.DASK_CLI and check_dask_subcommand. What remained is the part the issue's reproduction is really about: nothing runs the program, and nothing checks the options at all.

What this adds

Two test classes that ask the installed programs themselves, alongside (not instead of) the existing mocked tests.

Each check takes its command from the code that builds it for a real fit — setup_cluster, srun_worker_command, and the newly named Cluster.dask_scheduler_command — so no argument list is written down a second time. Locating cluster.DASK_CLI inside each command is itself the check that every worker-launch command goes through the one place that decides how dask is invoked.

The checks then confirm that this interpreter's dask CLI runs, that it still has the ssh, scheduler and worker subcommands, and that each subcommand's --help still declares every option PyBNF passes — so a renamed option fails as loudly as a renamed command. --nworkers is itself a survivor of exactly that, having been --nprocs until distributed removed the old name.

dask ssh gets the stricter treatment: the whole command PyBNF builds is handed to dask with --help appended, so dask does the parsing and refuses an unknown option with a non-zero exit. scheduler and worker cannot be asked that way — they forward unrecognized arguments to preload modules rather than refusing them — so for those the help screen is the witness, read from the option column alone (because dask worker --help names --nworkers in the prose describing three other options).

The same questions are asked of srun and scontrol, skipped wherever SLURM is absent. That does not make them dead weight: PyBNF's tests are run on clusters, which is the one place a renamed srun option can be caught before a fit walks into it — --cpus-per-task in particular is load-bearing, since without it a task confines every worker it forks to one CPU. srun's help is not laid out by click and its layout cannot be checked from a developer machine, so its options are matched as whole words: weaker, but still fatal to an option whose name has left the screen entirely.

Verification

Every check was confirmed to go red by injecting the failure it exists to catch:

injected change	result
--nworkers → --nprocs	3 fail, incl. dask's own No such option '--nprocs'
DASK_CLI → ['dask-ssh'] (the exact #615 regression)	17 fail — previously 0
srun's --cpus-per-task renamed (stand-in SLURM)	fails, naming the option
scontrol absent on a SLURM system	fails, naming the program

The whole-word matcher was also checked not to confuse --ntasks with --ntasks-per-node.

On the single named value

#619 asked that the command name live in one place instead of seven. cluster.DASK_CLI already existed; the test file now spells the invocation out exactly once, in DASK, and compares it against cluster.DASK_CLI in a single test.

The remaining full-argv oracles build on that one constant rather than repeating the literal. They are deliberately not changed to import DASK_CLI wholesale: a mocked test that asserts argv == cluster.DASK_CLI + [...] agrees with the module by construction and can no longer pin anything. The reality checks above are what make that spelled-out copy safe to keep.

Test plan

• tests/test_cluster.py: 132 passed, 2 skipped (the SLURM class, correctly skipping with no srun present).
• Full suite at the branch point vs. this branch, same environment: 4472 → 4481 passed, 23 → 25 skipped, 0 failed both times. The delta is exactly the new tests (9 running + 2 skipped); no test that passed before fails now.
• ruff@0.15.14 check . clean; sphinx-build -W --keep-going succeeds.