What happens
On a cluster where salloc opens its shell on the login node, starting a fit with the srun launcher fails to bring up any workers. srun refuses the step and the run stops:
srun: error: Unable to create step for job NNNNN: More processors requested than permitted
This is the launcher added in #614 and refined in #616 and #617. It is meant to be run from the shell that holds the allocation, which the cluster documentation says to open with salloc. On this cluster, and on many others, that shell runs on the login node while the allocation itself is held on a separate compute node.
Why it happens
The launcher asks SLURM for more CPUs per task than the job was granted.
In pybnf/cluster.py, srun_worker_layout first calls per_node_cpus, which reads $SLURM_JOB_CPUS_PER_NODE and gets the right per-node count. That variable is set correctly even on the login node. But the equal-size path then discards that count and calls srun_worker_command(scheduler_file, node_count, None), which re-derives the number on its own through cpus_per_node.
cpus_per_node prefers $SLURM_CPUS_ON_NODE. That variable is empty on the login node, because SLURM only sets it inside a step running on an allocated node. So cpus_per_node falls through to dask.system.CPU_COUNT, which reports the login node's own processors. The login node is large, so the launcher asks for far more CPUs per task than the allocation holds, and SLURM refuses the step.
The number that fails here is the same $SLURM_CPUS_ON_NODE the #616 fix chose to prefer. That choice is right for counting workers the SSH launcher starts on other machines, but it does not describe the allocation when the launching process is on the login node.
How to reproduce
On a cluster whose login node has more cores than the allocation, and whose salloc shell is the login node:
salloc -N1 -n20
# salloc returns a shell on the login node, holding the allocation on a compute node
pybnf -c job.conf -t slurm-srun
# srun: error: Unable to create step ...: More processors requested than permitted
Setting the variable the launcher looks for, to the number the job was granted, confirms the cause. With it set, the workers start and the fit runs:
export SLURM_CPUS_ON_NODE=20
pybnf -c job.conf -t slurm-srun
Suggested fix
srun_worker_layout already has the correct per-node counts from per_node_cpus before it decides which command to build. In the equal-size path it could pass that count on rather than letting srun_worker_command re-derive it. srun_worker_command_for_group already takes an explicit CPU count and does not consult cpus_per_node, so the equal-size case could route through it with counts[0], the same way the mixed-size case already does. That keeps the correct number, which per_node_cpus got from $SLURM_JOB_CPUS_PER_NODE, and which is right even when the launching process is on the login node.
Environment
PyBNF 1.7.0, SLURM, launched with -t slurm-srun from an interactive salloc shell on the login node.
What happens
On a cluster where salloc opens its shell on the login node, starting a fit with the srun launcher fails to bring up any workers. srun refuses the step and the run stops:
This is the launcher added in #614 and refined in #616 and #617. It is meant to be run from the shell that holds the allocation, which the cluster documentation says to open with salloc. On this cluster, and on many others, that shell runs on the login node while the allocation itself is held on a separate compute node.
Why it happens
The launcher asks SLURM for more CPUs per task than the job was granted.
In pybnf/cluster.py, srun_worker_layout first calls per_node_cpus, which reads $SLURM_JOB_CPUS_PER_NODE and gets the right per-node count. That variable is set correctly even on the login node. But the equal-size path then discards that count and calls srun_worker_command(scheduler_file, node_count, None), which re-derives the number on its own through cpus_per_node.
cpus_per_node prefers $SLURM_CPUS_ON_NODE. That variable is empty on the login node, because SLURM only sets it inside a step running on an allocated node. So cpus_per_node falls through to dask.system.CPU_COUNT, which reports the login node's own processors. The login node is large, so the launcher asks for far more CPUs per task than the allocation holds, and SLURM refuses the step.
The number that fails here is the same $SLURM_CPUS_ON_NODE the #616 fix chose to prefer. That choice is right for counting workers the SSH launcher starts on other machines, but it does not describe the allocation when the launching process is on the login node.
How to reproduce
On a cluster whose login node has more cores than the allocation, and whose salloc shell is the login node:
Setting the variable the launcher looks for, to the number the job was granted, confirms the cause. With it set, the workers start and the fit runs:
Suggested fix
srun_worker_layout already has the correct per-node counts from per_node_cpus before it decides which command to build. In the equal-size path it could pass that count on rather than letting srun_worker_command re-derive it. srun_worker_command_for_group already takes an explicit CPU count and does not consult cpus_per_node, so the equal-size case could route through it with counts[0], the same way the mixed-size case already does. That keeps the correct number, which per_node_cpus got from $SLURM_JOB_CPUS_PER_NODE, and which is right even when the launching process is on the login node.
Environment
PyBNF 1.7.0, SLURM, launched with -t slurm-srun from an interactive salloc shell on the login node.