Split an explicit parallel_count across a mixed allocation by machine size (#643)

[wshlavacek]

Fixes #643.

On a cluster whose machines are not all the same size, setting parallel_count could keep the slurm-srun launcher from starting any worker. parallel_count is a total number of workers split evenly across the machines, and the launcher asked SLURM in a single job step for the even per-machine share on every machine. On a mixed allocation that share can be more than a smaller machine was granted, and SLURM refuses the whole step with "More processors requested than permitted", so the fit stops before it starts.

Repro: an allocation of one 96-CPU machine and one 40-CPU machine with parallel_count set to 136. The even share is 68 per machine. Launched from the 96-CPU machine the cap is 96, so the step asks for 68 CPUs on both machines, and the 40-CPU machine cannot supply them. No worker starts.

The automatic path (parallel_count unset) already avoids this. Issue #617 gave it one srun step per distinct machine size, each asking only for what those machines hold. Only the parallel_count override was left as a single even-split step, which ADR-0124 recorded and asked to track separately, because a fix changes what the config key means on a mixed allocation.

What this changes:

On a mixed allocation, parallel_count is now split in proportion to each machine's granted CPUs, started as one srun step per distinct size, the same way the automatic path works. Each step asks only for what its own machines hold, so SLURM accepts every step. On the repro above this becomes one worker per CPU: 96 workers on the large machine, 40 on the small one, both fully bound.

The rule is the exact generalization of the even split. On a same-size allocation each machine's share works out to ceil(parallel_count divided by the machine count), the number the even split already produces, so a homogeneous run builds exactly the single step it built before. A homogeneous allocation, with or without parallel_count, is byte for byte unchanged, and so is the automatic mixed path from #617.

ADR-0126 records the decision and supersedes the deferral ADR-0124 recorded. ADR-0124 gets a one-line pointer to it.

Tests: the srun launcher tests use stand-ins for srun and scontrol, and the constructed srun argument list is the oracle, as it was for #614, #617, and #642. The suite now checks the proportional split on a mixed allocation, the reported failure case, an oversubscribed parallel_count that stays bound, and that a homogeneous allocation with parallel_count is still one even-split step. All cluster tests pass with:

uv run --extra tests python -m pytest tests/test_cluster.py -q