What happens
How many processors a fit actually keeps busy is decided by the fitting settings, mainly population_size, and not by how many processors the user reserved. If those two numbers do not match, the extra processors sit idle for the whole run and nothing says so.
A user can reserve several machines, wait in the queue, and use a fraction of what they reserved without ever finding out.
An example
The short test problem in tests/full_tests/T1-ssprop uses population_size = 12. On two machines of ordinary size, that is far fewer simulations running at once than there are worker processes available. Most of the reserved capacity would do nothing.
The opposite mismatch is also silent. A population much larger than the number of workers buys no extra speed and only makes work queue up.
Why it happens
PyBNF submits one job for each parameter set the algorithm proposes at the start, then submits roughly one new job each time a job finishes. The amount of work in progress therefore follows the size of the population, and has no relation to the size of the cluster.
The relevant code is in pybnf/algorithms/base.py, in the run method and in _drain_job_pool.
Suggested fix
After connecting to the cluster and submitting the first set of jobs, compare the number of jobs in progress with the number of workers that have connected. Dask reports the second number. Print a clear warning when the two differ by a large margin in either direction, naming both numbers and the setting that controls the first.
Some algorithms wait for every simulation in a round to finish before starting the next round, so idle time is expected with those. Say so in the same message when one of those is in use, to avoid sending the user looking for a fault that is not there.
Record both numbers in the log for every run across machines, so a finished run can be looked at afterwards.
Status
This has not yet been seen on a real cluster. Every multi-machine run we attempted stopped at startup for the reasons given in issues #614 and #615, so no fit ran long enough to measure. The reasoning above comes from reading the code. Confirming it needs a working multi-machine run.
What happens
How many processors a fit actually keeps busy is decided by the fitting settings, mainly population_size, and not by how many processors the user reserved. If those two numbers do not match, the extra processors sit idle for the whole run and nothing says so.
A user can reserve several machines, wait in the queue, and use a fraction of what they reserved without ever finding out.
An example
The short test problem in tests/full_tests/T1-ssprop uses population_size = 12. On two machines of ordinary size, that is far fewer simulations running at once than there are worker processes available. Most of the reserved capacity would do nothing.
The opposite mismatch is also silent. A population much larger than the number of workers buys no extra speed and only makes work queue up.
Why it happens
PyBNF submits one job for each parameter set the algorithm proposes at the start, then submits roughly one new job each time a job finishes. The amount of work in progress therefore follows the size of the population, and has no relation to the size of the cluster.
The relevant code is in pybnf/algorithms/base.py, in the run method and in _drain_job_pool.
Suggested fix
After connecting to the cluster and submitting the first set of jobs, compare the number of jobs in progress with the number of workers that have connected. Dask reports the second number. Print a clear warning when the two differ by a large margin in either direction, naming both numbers and the setting that controls the first.
Some algorithms wait for every simulation in a round to finish before starting the next round, so idle time is expected with those. Say so in the same message when one of those is in use, to avoid sending the user looking for a fault that is not there.
Record both numbers in the log for every run across machines, so a finished run can be looked at afterwards.
Status
This has not yet been seen on a real cluster. Every multi-machine run we attempted stopped at startup for the reasons given in issues #614 and #615, so no fit ran long enough to measure. The reasoning above comes from reading the code. Confirming it needs a working multi-machine run.