Warn when a fit uses far fewer processors than were reserved (#621)

[wshlavacek]

Closes #621.

How many simulations a fit runs at once is decided by its settings, mainly population_size, and not by how many processors the user reserved. When the two do not match, the extra processors can sit idle for the whole run and nothing said so. A user could reserve several machines, wait in the queue, and quietly use a fraction of what they reserved.

What this changes:

After the fit submits its first set of jobs, it compares how many jobs are running with how many workers connected to the cluster. It logs both numbers so a finished run can be looked at afterwards, and it prints a warning when the two differ by a large margin in either direction. A factor of two is the margin. The warning names both numbers and points at population_size, and suggests either changing population_size or changing how many processors are reserved.

Some methods run one generation at a time and wait for the whole generation to finish before starting the next, so some idle time toward the end of each generation is normal with them. Differential evolution, CMA-ES, and scatter search are marked as such, and their message says the idle time is expected. This is meant to save the user from looking for a fault that is not there.

Only cluster runs are reported. A local run's worker count is exactly what the user asked for through parallel_count, so there is nothing to compare it against. Reading the worker count from dask is wrapped so a failure there logs and skips the report rather than stopping a fit.

Confirmed on a real cluster:

Issue #621 said this had never been seen on a real multi-machine run, because the runs we tried stopped at start up for the reasons in #614 and #615. Those two are now fixed, so the run went through. A differential evolution fit with population_size = 6 was run across two machines using the srun start up path. SLURM granted both machines in full, which was 64 processors each, so 128 workers connected. The fit printed:

Warning: The fit starts with only 6 job(s) running but 128 worker(s) connected, so about 122 worker(s) will sit idle. How many jobs run at once is set by the fitting settings, mainly population_size, not by how many processors were reserved. Consider raising population_size or reserving fewer processors. This fit runs one generation at a time and waits for all of it to finish before starting the next, so some idle time toward the end of each generation is expected.

So the case #621 described is real. Reserving two machines and setting a small population left 122 of 128 processors doing nothing for the whole run, and now the fit says so at the start.

Testing:

New unit tests in tests/test_run_loop.py cover the local case (nothing reported), a well matched cluster run (both numbers logged, no warning), too many workers (idle warning that names population_size and prints to the console), too many jobs (queue warning), the generational note in both the warning and the well matched case, zero workers, an unreadable worker count, and one end to end run through the main loop. The full run loop, cluster, optimizer integration, differential evolution, and scatter search test files pass.