From dc0c6a547ae7eaf35944c82491680e71e57c9568 Mon Sep 17 00:00:00 2001 From: Bill Hlavacek Date: Fri, 21 Aug 2026 14:18:56 -0600 Subject: [PATCH] docs(cluster): the cluster page names commands that exist, tells a user which runs log in anywhere, and says how to size a fit to the processors reserved (#622) `docs/cluster.rst` is the only guidance a user has for running PyBNF on a cluster, and parts of it described a program that no longer exists, answered a failure with advice that cannot fix it, and said nothing about the setting that decides whether an allocation is used at all. The command names were corrected when the launcher was (#615, #631); this is the rest of #622. **Which ways of starting a run log in to other machines.** The old "Troubleshooting: SSH access to nodes" treated every multi-machine failure as a missing key. On the cluster in #614 keys were not the obstacle: logging in by hand already worked, and the run still failed, because `dask ssh` does not run `ssh` -- it logs in through paramiko, which can offer a public key or a typed password and nothing else, so a cluster that authenticates its nodes by host-based or Kerberos SSH refuses a login the user can make by hand. A table now says plainly which of the five ways of starting a run involves a login at all (`-t slurm` and `scheduler_node`/`worker_nodes` do; `-t slurm-srun`, `-s cluster.json` and a single-machine run do not), and the section hands the reader the login PyBNF actually attempts, as a one-line paramiko connection that mirrors `distributed.deploy.old_ssh` argument for argument. Its three outcomes are separated: it succeeds, so `-t slurm` will work; it raises `AuthenticationException` where plain `ssh` succeeds, so no key will help and `-t slurm-srun` is the answer; or it asks for a password, which is the one case `ssh-keygen` fixes. **Sizing a run.** Nothing on the page connected the processors a user reserves to the number PyBNF can keep busy, which is the choice that decides whether a run is worth its queue time. A new section gives the two numbers -- the workers, which is what the job was granted, and the simulations in flight, which is `population_size` for almost every algorithm -- and a table of what each `fit_type` can actually run at once: `population_size` for the population methods and the samplers, `population_size` x (`population_size` - 1) for scatter search, min(`population_size`, N-1) per start for Simplex, one per start for Powell, two walks per parameter for profile likelihood, and none at all for `hmc`, whose chains never reach a worker. The wave arithmetic of a synchronized algorithm is stated (129 parameter sets on 128 workers costs what 256 would), and the shipped tcr example is walked through, since its `population_size = 9` is exactly the 72 processors its batch script reserves. **Where simulation files are written.** The `simulation_dir` advice read as general when it is not: on the ordinary network filesystem most clusters give you the default is right, and the key is for a parallel filesystem such as Lustre or GPFS. The new section says which case is which, and adds the requirement that made it worth writing down -- the directory has to be visible from the node PyBNF runs on, because that is where the best fit's output is copied from when `delete_old_files = 0`. **Example batch scripts.** The nine scripts that told the reader to load Anaconda for "Python 3.5" now carry a marked line to edit for a Python 3.11 or newer environment. The tcr and tlbr scripts additionally reserve processors that match their own configs (4 x 18 = 72 for scatter search with a reference set of 9; 5 x 30 = 150 for 250 parameter sets x 3 smoothing replicates), so the two examples the page points at demonstrate the sizing section rather than contradicting it. Also corrected in passing: `docs/algorithms.rst` claimed Simplex fans out to `parallel_count` processors and could use N+1 of them. It uses min(`population_size`, N-1) per start, times `n_starts` concurrent starts, and has since #498. Prose only -- no key changes meaning and no code is touched. `sphinx-build -W --keep-going` is clean. Signed-off-by: Bill Hlavacek --- docs/algorithms.rst | 4 +- docs/cluster.rst | 115 +++++++++++++++++-- docs/config_keys.rst | 12 +- examples/Degranulation_aMCMC/degran_mcmc.sh | 9 +- examples/HIVdynamics_aMCMC/pt303/pt303.sh | 9 +- examples/HIVdynamics_aMCMC/pt403/pt403.sh | 9 +- examples/HIVdynamics_aMCMC/pt409/pt409.sh | 9 +- examples/LinearRegression_aMCMC/linear_am.sh | 9 +- examples/LinearRegression_aMCMC/linear_mh.sh | 9 +- examples/degranulation/degran_mcmc.sh | 9 +- examples/tcr/tcr_batch.sh | 26 +++-- examples/tlbr/tlbr_batch.sh | 28 +++-- 12 files changed, 185 insertions(+), 63 deletions(-) diff --git a/docs/algorithms.rst b/docs/algorithms.rst index 2d3dc3e6b..c1ccc64f7 100644 --- a/docs/algorithms.rst +++ b/docs/algorithms.rst @@ -614,7 +614,7 @@ Simplex is a local search algorithm that operates solely on objective evaluation Parallelization ^^^^^^^^^^^^^^^ -The PyBNF Simplex implementation is parallel and synchronous. Synchronization is required at the end of every iteration. Parallelization is achieved by simultaneously evaluating a subset of the N+1 points in the simplex. Therefore, this parallelization can take advantage of at most N+1 processors, where N is the number of free parameters. +The PyBNF Simplex implementation is parallel and synchronous. Synchronization is required at the end of every iteration. Parallelization is achieved by simultaneously evaluating a subset of the N+1 points in the simplex, where N is the number of free parameters. One simplex evaluates min(``population_size``, N-1) points per iteration -- never fewer than 1 -- so a single search can take advantage of at most that many processors. Running several starts concurrently with ``n_starts`` multiplies it: ``n_starts`` x min(``population_size``, N-1) points are in flight at once. Implementation details @@ -631,7 +631,7 @@ The initial simplex consists of N+1 points chosen deterministically based on the Illustration of the simplex algorithm, modifying point P on a 3-point simplex in 2 dimensions -Each iteration, we operate on the k worst points in the simplex, where k is the number of available processors (``parallel_count``). For each point P, we consider the hyperplane defined by the other N points in the simplex (blue line). Let d be the distance from P to the hyperplane. We evaluate point P\ :sub:`1` obtained by reflecting P through the hyperplane, to a distance of d \* ``simplex_reflect`` on the other side. Depending on the resulting objective value, we try another point in the second phase of the iteration. Three cases are possible. +Each iteration, we operate on the k worst points in the simplex, where k is min(``population_size``, N-1). For each point P, we consider the hyperplane defined by the other N points in the simplex (blue line). Let d be the distance from P to the hyperplane. We evaluate point P\ :sub:`1` obtained by reflecting P through the hyperplane, to a distance of d \* ``simplex_reflect`` on the other side. Depending on the resulting objective value, we try another point in the second phase of the iteration. Three cases are possible. 1) The new point is better than the current global minimum: We try a second point continuing in the same direction for a distance of d \* ``simplex_expansion`` away from the hyperplane (P\ :sub:`2,1`). 2) The new point is worse than the global minimum, but better than the next worst point in the simplex: We don't try a second point. diff --git a/docs/cluster.rst b/docs/cluster.rst index 3069d72de..6734e9a26 100644 --- a/docs/cluster.rst +++ b/docs/cluster.rst @@ -27,25 +27,56 @@ Load the appropriate Python environment Initiate a PyBNF fitting run, including the flag ``-t slurm`` +If you intend to use ``-t slurm-srun`` instead, skip the ``slogin`` step and run PyBNF from the shell ``salloc`` opened: a separate login into a node does not inherit the allocation, and that launcher needs it. See `Starting workers without SSH`_. + Batch ^^^^^ Write a shell script specifying the desired nodes and their properties `according to SLURM specifications `_. Be sure that your script includes loading the appropriate Python environment if this step is required for your cluster, and that your call to pybnf includes the flag ``-t slurm``. For an example shell script, see examples/tcr/tcr_batch.sh. Submit the batch job to the queueing system using the command ``sbatch script.sh`` where ``script.sh`` is the name of the shell script. -Troubleshooting: SSH access to nodes -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -The above instructions assume that PyBNF can access all allocated nodes via SSH. For some clusters, additional configuration is necessary to enable SSH access: use ``ssh_keygen`` (documented in many places, such as `here `__, or `here `__ for instructions specific to PyBNF's Dask scheduler) to set up SSH keys. +.. _sshlogin: + +Which ways of starting a run log in to other machines +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Not every way of running PyBNF on several machines logs in to them, so before troubleshooting an SSH problem, check whether the way you start your run involves an SSH login at all. + +.. list-table:: + :header-rows: 1 + :widths: 30 70 + + * - How the run is started + - Does PyBNF log in to the other machines? + * - ``-t slurm`` + - **Yes.** PyBNF runs ``dask ssh``, which opens an SSH connection from the node PyBNF is running on to every allocated node. + * - ``scheduler_node`` / ``worker_nodes`` (`Manual configuration with node names`_) + - **Yes** -- the same ``dask ssh`` launcher, over node names you supplied instead of ones read from SLURM. + * - ``-t slurm-srun`` (`Starting workers without SSH`_) + - **No.** SLURM starts the workers inside the allocation it already granted, and no credential is involved. + * - ``-s cluster.json`` (`Manual configuration with Dask`_) + - **No.** You start the scheduler and the workers yourself; PyBNF only connects to the scheduler. + * - A single-machine run + - There are no other machines. + +If you use one of the first two, the login must succeed without a password prompt, from the node PyBNF runs on to every other node of the job. + +**Test the login PyBNF makes, not the one you can make.** ``ssh othernode hostname`` succeeding proves little here, because it is not the login PyBNF attempts: ``dask ssh`` does not run ``ssh``. It connects with the paramiko library, which can offer a public key or a typed password and nothing else. From a node of your allocation, make the same connection paramiko will make:: + + python -c "import paramiko; c = paramiko.SSHClient(); \ + c.set_missing_host_key_policy(paramiko.AutoAddPolicy()); \ + c.connect('OTHERNODE'); print('paramiko login ok')" -To confirm that SSH keys are set up correctly, make sure that you are able to SSH into all allocated nodes without needing to enter a password. +* It prints ``paramiko login ok``: ``-t slurm`` can start your workers. +* It raises ``AuthenticationException`` while plain ``ssh`` to the same node succeeds: your cluster authenticates its nodes to each other by a method paramiko cannot use -- most often host-based or Kerberos (GSSAPI) SSH. **Creating SSH keys cannot fix this**, because the cluster is not asking for a key. Use `Starting workers without SSH`_. +* It asks for a password, or fails in the same way plain ``ssh`` does: this is the case SSH keys do fix. Create a key pair with ``ssh-keygen`` (documented in many places, such as `here `__) and append the public half to ``~/.ssh/authorized_keys``. Where the nodes share your home directory, that covers all of them at once. Then run the check above again. -Setting up SSH keys does not help on every cluster. If your cluster's nodes authenticate to each other by host-based or Kerberos SSH, use `Starting workers without SSH`_ instead; if SSH access is not possible for some other reason, you can also fall back on `Manual configuration with Dask`_. +If SSH cannot be made to work for some other reason, `Starting workers without SSH`_ and `Manual configuration with Dask`_ both avoid it entirely. .. _srun: Starting workers without SSH ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -``-t slurm`` starts the workers by logging in to each allocated node over SSH, using dask's ``dask ssh`` command. That command logs in with the paramiko library, which offers only two ways of authenticating: a public key, or a password. Clusters commonly use two others: +As `Which ways of starting a run log in to other machines`_ describes, ``-t slurm`` starts the workers by logging in to each allocated node with ``dask ssh``, which authenticates through paramiko. Clusters commonly use two methods paramiko cannot offer: * **host-based authentication**, where the machines are configured to trust each other and no user credential is involved. paramiko does not support this at all. * **Kerberos (GSSAPI)**. paramiko can do this, but dask never turns it on. @@ -60,16 +91,17 @@ Pass ``-t slurm-srun`` instead (or set ``cluster_type = slurm-srun``) to start t Run PyBNF from the shell that holds the allocation: the one ``salloc`` opened, or your ``sbatch`` script. A separate login into one of the allocated nodes does not inherit the allocation, and ``srun`` would then queue a new job rather than start the workers; PyBNF refuses to start in that case instead of appearing to hang. For the same reason, PyBNF should be the only job step running in the allocation, since a concurrent second ``srun`` can leave the workers waiting for resources. -An example batch script, the ``-t slurm`` one with a single word changed:: +An example batch script -- ``examples/tcr/tcr_batch.sh`` with a single word changed:: #!/bin/bash #SBATCH --nodes=4 - #SBATCH --mincpus=32 + #SBATCH --mincpus=18 #SBATCH --time=1-00:00:00 #SBATCH --job-name=pybnf - # Your cluster may require loading a module to get the right Python environment. - module load anaconda/Anaconda3 + # EDIT THIS LINE for your cluster: load a module (or activate a virtual + # environment) that provides Python 3.11 or newer with PyBNF installed. + module load python/3.11 pybnf -c tcr-ss.conf -t slurm-srun -o @@ -93,6 +125,69 @@ Which number was used, and which of the three it came from, is written to the lo Setting ``parallel_count`` overrides all of this with a total number of worker processes over all nodes, divided evenly among them; the log then names ``parallel_count`` as the source. Nodes of different sizes still get equal shares. +.. _sizing: + +Sizing a run +------------ + +Reserving processors does not by itself make PyBNF use them. Two numbers decide how many are busy, and they are set in different places: + +* **How many worker processes exist.** By default, the number of CPUs your job was granted (nodes times CPUs per node), or ``parallel_count`` if you set it. See `How many workers run on each node`_. +* **How many simulations the algorithm can have running at once.** For almost every algorithm this is ``population_size``. + +Make the second at least as large as the first. ``population_size = 50`` on a 128-processor allocation leaves 78 processors idle for the whole fit, and the queue was waited out for all 128 -- so processors the algorithm cannot use cost twice, once in the wait for a bigger allocation and again in the share of it that sits doing nothing. + +For a **synchronized** algorithm -- the Parallelization column of the table in :ref:`Algorithms ` says which -- the population is evaluated in waves, and an iteration costs ``ceil(population_size / workers)`` waves however uneven the last one is. Going from 128 parameter sets to 129 on 128 workers therefore nearly doubles the time per iteration in exchange for one extra parameter set, so prefer a ``population_size`` that is a whole multiple of the worker count. An **asynchronous** algorithm (``ade`` or ``pso``, or ``de`` with ``islands`` greater than 1) starts a new simulation the moment one finishes and so has no such cliff, which also makes it the better choice when simulation times vary a lot; it still needs ``population_size`` at least the worker count to fill the machine. + +How many simulations each algorithm runs at once +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. list-table:: + :header-rows: 1 + :widths: 30 70 + + * - ``fit_type`` + - Simulations that can run at the same time + * - ``de``, ``ade``, ``pso``, ``cmaes``, ``dream``, ``p_dream``, ``mh``, ``am``, ``sa``, ``pt`` + - ``population_size``. For ``pt`` that is the replicas at all temperatures together; for the Markov chain samplers it is the number of independent chains. + * - ``ss`` + - ``population_size`` x (``population_size`` - 1), since every parent-helper pair is a simulation. A reference set of 9 fills 72 processors. + * - ``sim`` + - min(``population_size``, N - 1) per start, never fewer than 1, where N is the number of free parameters; times ``n_starts`` concurrent starts. + * - ``powell`` + - One per start, so ``n_starts``: each line search is serial by construction. + * - ``trf``, ``lbfgs``, ``gntr`` + - ``population_size``, which the gradient optimizers use as their number of concurrent starts. + * - ``pl`` + - Two directional walks per profiled parameter (one per direction), capped by ``profile_likelihood_max_parallel`` -- ``0``, the default, runs all of them at once. + * - ``hmc`` + - None. Its chains are an in-process numeric loop rather than dask jobs (and it runs only on analytical models), so extra nodes do not help. + +Multiply any of these by ``smoothing`` and by ``parallelize_models``, if you set them: every replicate and every model partition is a separate job. ``n_starts`` adds nothing for the metaheuristics (``de``, ``ade``, ``ss``, ``pso``), whose starts run one after another rather than at the same time. + +The processors an algorithm cannot use are worth reserving only for the memory attached to them. Each worker is a separate process holding its own copy of the models, so a memory-hungry model may need ``parallel_count`` set below the CPU count rather than a bigger population. + +A worked example +^^^^^^^^^^^^^^^^ + +``examples/tcr/tcr-ss.conf`` runs scatter search with ``population_size = 9``, which is 9 x 8 = 72 simulations per iteration. Its batch script therefore reserves 72 processors (4 nodes of 18), and its ``parallel_count = 72`` states the same number a second time. Reserving 144 instead would not make that fit finish any sooner: the reference set, not the allocation, sets the work. Scatter search fills 72, 90, 110, 132 or 156 processors as ``population_size`` goes 9, 10, 11, 12, 13, so with this algorithm it is the allocation that should be chosen to match the population. + +.. _simdir: + +Where simulation files are written +---------------------------------- + +Every simulation runs in its own directory under ``output_dir``, so a large fit creates and deletes a great many small files. On the ordinary shared network filesystem most clusters give you -- an NFS home or project space -- that is unremarkable, and the default (simulations beside the results) is the right setting. Leave ``simulation_dir`` unset. + +It is worth changing on a **parallel filesystem** such as Lustre or GPFS, which is tuned for a few large streaming reads and writes and handles storms of small-file metadata operations worst; some sites also meter it. There, set ``simulation_dir`` to a path on storage better suited to the traffic, and only the results are written to the parallel filesystem:: + + simulation_dir = /scratch/username/pybnf + +Whatever you point it at must exist and be writable on every node, since it is the workers that write there. Keep it on shared storage: PyBNF looks for the best fit's simulation output under ``simulation_dir`` from the node it is running on, so with node-local storage and ``delete_old_files = 0``, the copy of the best-fit ``gdat`` files into ``Results/`` cannot find them. + +If you do not know which kind of filesystem your ``output_dir`` lives on, ask your administrators rather than guessing. Setting this key on a cluster that does not need it gains nothing, and pointing it somewhere the compute nodes cannot write turns a working fit into a failing one. + + TORQUE/PBS ---------- Not yet implemented. Please refer to Manual configuration below diff --git a/docs/config_keys.rst b/docs/config_keys.rst index b6a7eb8d2..6db0f0a90 100644 --- a/docs/config_keys.rst +++ b/docs/config_keys.rst @@ -632,7 +632,8 @@ Required Keys **population_size** The number parameter sets to maintain in a single iteration of the algorithm. See algorithm descriptions for more - information. + information. This is also the key that decides how many simulations can run at the same time, so on a cluster it + should be chosen against the number of processors reserved -- see :ref:`Sizing a run `. Example: @@ -1031,9 +1032,12 @@ Parallel Computing **simulation_dir** Optional setting for a different directory where we should save (or temporarily store) simulation output. Usually - not necessary to set separately from `output_dir`. However, if you are running on a cluster with a Lustre filesystem, - you may want to set this to a different disk to avoid excessive reads and writes to the Lustre disk. - + not necessary to set separately from `output_dir`, and on an ordinary shared network filesystem there is no reason + to. It is worth setting when `output_dir` is on a parallel filesystem such as Lustre or GPFS, which handles the many + small files a fit creates and deletes worst; point this key at storage better suited to that traffic and only the + results are written to the parallel filesystem. It must exist and be writable on every node, and should be on shared + storage -- see :ref:`Where simulation files are written `. + Default: Use the same directory as `output_dir`. Example: diff --git a/examples/Degranulation_aMCMC/degran_mcmc.sh b/examples/Degranulation_aMCMC/degran_mcmc.sh index 422f9e9cb..e24920c84 100644 --- a/examples/Degranulation_aMCMC/degran_mcmc.sh +++ b/examples/Degranulation_aMCMC/degran_mcmc.sh @@ -1,7 +1,7 @@ #!/bin/bash # Example batch script for running MCMC for the degranulation model on a SLURM cluster -# set the number of nodesT +# set the number of nodes #SBATCH --nodes=1 # set the number of cpus per node. @@ -15,10 +15,11 @@ # set name of job #SBATCH --job-name=pybnf -# Enable Anaconda Python 3.5 -# Your cluster might require something different here, or might not require anything. +# EDIT THIS LINE for your cluster: load a module (or activate a virtual environment) +# that provides Python 3.11 or newer with PyBNF installed. Some clusters need nothing +# here at all. module purge -module load anaconda3/2020.02 +module load python/3.11 # Run PyBNF pybnf -c mcmc.conf -t SLURM -o diff --git a/examples/HIVdynamics_aMCMC/pt303/pt303.sh b/examples/HIVdynamics_aMCMC/pt303/pt303.sh index 2c4ee1a2f..413b4093b 100644 --- a/examples/HIVdynamics_aMCMC/pt303/pt303.sh +++ b/examples/HIVdynamics_aMCMC/pt303/pt303.sh @@ -1,7 +1,7 @@ #!/bin/bash # Example batch script for running MCMC for the degranulation model on a SLURM cluster -# set the number of nodesT +# set the number of nodes #SBATCH --nodes=1 # set the number of cpus per node. @@ -15,10 +15,11 @@ # set name of job #SBATCH --job-name=pybnf -# Enable Anaconda Python 3.5 -# Your cluster might require something different here, or might not require anything. +# EDIT THIS LINE for your cluster: load a module (or activate a virtual environment) +# that provides Python 3.11 or newer with PyBNF installed. Some clusters need nothing +# here at all. module purge -module load anaconda3/2020.02 +module load python/3.11 # Run PyBNF pybnf -c pt303.conf -t SLURM -o diff --git a/examples/HIVdynamics_aMCMC/pt403/pt403.sh b/examples/HIVdynamics_aMCMC/pt403/pt403.sh index d634609b5..3e3f3728b 100644 --- a/examples/HIVdynamics_aMCMC/pt403/pt403.sh +++ b/examples/HIVdynamics_aMCMC/pt403/pt403.sh @@ -1,7 +1,7 @@ #!/bin/bash # Example batch script for running MCMC for the degranulation model on a SLURM cluster -# set the number of nodesT +# set the number of nodes #SBATCH --nodes=1 # set the number of cpus per node. @@ -15,10 +15,11 @@ # set name of job #SBATCH --job-name=pybnf -# Enable Anaconda Python 3.5 -# Your cluster might require something different here, or might not require anything. +# EDIT THIS LINE for your cluster: load a module (or activate a virtual environment) +# that provides Python 3.11 or newer with PyBNF installed. Some clusters need nothing +# here at all. module purge -module load anaconda3/2020.02 +module load python/3.11 # Run PyBNF pybnf -c pt403.conf -t SLURM -o \ No newline at end of file diff --git a/examples/HIVdynamics_aMCMC/pt409/pt409.sh b/examples/HIVdynamics_aMCMC/pt409/pt409.sh index 3170b02f4..7b9e6ac54 100644 --- a/examples/HIVdynamics_aMCMC/pt409/pt409.sh +++ b/examples/HIVdynamics_aMCMC/pt409/pt409.sh @@ -1,7 +1,7 @@ #!/bin/bash # Example batch script for running MCMC for the degranulation model on a SLURM cluster -# set the number of nodesT +# set the number of nodes #SBATCH --nodes=1 # set the number of cpus per node. @@ -15,10 +15,11 @@ # set name of job #SBATCH --job-name=pybnf -# Enable Anaconda Python 3.5 -# Your cluster might require something different here, or might not require anything. +# EDIT THIS LINE for your cluster: load a module (or activate a virtual environment) +# that provides Python 3.11 or newer with PyBNF installed. Some clusters need nothing +# here at all. module purge -module load anaconda3/2020.02 +module load python/3.11 # Run PyBNF pybnf -c pt409.conf -t SLURM -r \ No newline at end of file diff --git a/examples/LinearRegression_aMCMC/linear_am.sh b/examples/LinearRegression_aMCMC/linear_am.sh index fe48a8d97..c01fc8ae6 100644 --- a/examples/LinearRegression_aMCMC/linear_am.sh +++ b/examples/LinearRegression_aMCMC/linear_am.sh @@ -1,7 +1,7 @@ #!/bin/bash # Example batch script for running MCMC for the degranulation model on a SLURM cluster -# set the number of nodesT +# set the number of nodes #SBATCH --nodes=1 # set the number of cpus per node. @@ -15,10 +15,11 @@ # set name of job #SBATCH --job-name=pybnf_am -# Enable Anaconda Python 3.5 -# Your cluster might require something different here, or might not require anything. +# EDIT THIS LINE for your cluster: load a module (or activate a virtual environment) +# that provides Python 3.11 or newer with PyBNF installed. Some clusters need nothing +# here at all. module purge -module load anaconda3/2020.02 +module load python/3.11 # Run PyBNF pybnf -c linear_am.conf -t SLURM -o diff --git a/examples/LinearRegression_aMCMC/linear_mh.sh b/examples/LinearRegression_aMCMC/linear_mh.sh index 26f0a5205..7fc7d57f0 100644 --- a/examples/LinearRegression_aMCMC/linear_mh.sh +++ b/examples/LinearRegression_aMCMC/linear_mh.sh @@ -1,7 +1,7 @@ #!/bin/bash # Example batch script for running MCMC for the degranulation model on a SLURM cluster -# set the number of nodesT +# set the number of nodes #SBATCH --nodes=1 # set the number of cpus per node. @@ -15,10 +15,11 @@ # set name of job #SBATCH --job-name=pybnf_mh -# Enable Anaconda Python 3.5 -# Your cluster might require something different here, or might not require anything. +# EDIT THIS LINE for your cluster: load a module (or activate a virtual environment) +# that provides Python 3.11 or newer with PyBNF installed. Some clusters need nothing +# here at all. module purge -module load anaconda3/2020.02 +module load python/3.11 # Run PyBNF pybnf -c linear_mh.conf -t SLURM -o diff --git a/examples/degranulation/degran_mcmc.sh b/examples/degranulation/degran_mcmc.sh index 3e8080688..6f7e1d986 100644 --- a/examples/degranulation/degran_mcmc.sh +++ b/examples/degranulation/degran_mcmc.sh @@ -1,7 +1,7 @@ #!/bin/bash # Example batch script for running MCMC for the degranulation model on a SLURM cluster -# set the number of nodesT +# set the number of nodes #SBATCH --nodes=4 # set the number of cpus per node. @@ -13,9 +13,10 @@ # set name of job #SBATCH --job-name=pybnf -# Enable Anaconda Python 3.5 -# Your cluster might require something different here, or might not require anything. -module load anaconda/Anaconda3 +# EDIT THIS LINE for your cluster: load a module (or activate a virtual environment) +# that provides Python 3.11 or newer with PyBNF installed. Some clusters need nothing +# here at all. +module load python/3.11 # Run PyBNF pybnf -c mcmc.conf -t SLURM -o diff --git a/examples/tcr/tcr_batch.sh b/examples/tcr/tcr_batch.sh index 7028a764f..28c69179e 100644 --- a/examples/tcr/tcr_batch.sh +++ b/examples/tcr/tcr_batch.sh @@ -1,11 +1,16 @@ #!/bin/bash -# Example batch script for running tcr example on a SLURM cluster +# Example batch script for running the tcr example on a SLURM cluster -# set the number of nodesT +# The processors reserved here are sized to the fit in tcr-ss.conf: scatter search +# with population_size=9 runs 9 x 8 = 72 simulations per iteration, so 4 nodes of 18 +# processors keeps every one of them busy. See "Sizing a run" in the PyBNF +# documentation before changing either number. + +# set the number of nodes #SBATCH --nodes=4 -# set the number of cpus per node. -#SBATCH --mincpus=32 +# set the number of cpus per node +#SBATCH --mincpus=18 # set max wallclock time for the entire fitting job (1 day) #SBATCH --time=1-00:00:00 @@ -13,9 +18,12 @@ # set name of job #SBATCH --job-name=pybnf -# Enable Anaconda Python 3.5 -# Your cluster might require something different here, or might not require anything. -module load anaconda/Anaconda3 +# EDIT THIS LINE for your cluster: load a module (or activate a virtual environment) +# that provides Python 3.11 or newer with PyBNF installed. Some clusters need nothing +# here at all. +module load python/3.11 -# Run PyBNF -pybnf -c tcr-ss.conf -t SLURM -o +# Run PyBNF. Use "-t slurm-srun" instead if your cluster's nodes cannot be logged +# into over SSH with a key or a password; see "Which ways of starting a run log in to +# other machines" in the PyBNF documentation. +pybnf -c tcr-ss.conf -t slurm -o diff --git a/examples/tlbr/tlbr_batch.sh b/examples/tlbr/tlbr_batch.sh index d536c9d9b..5c31b145c 100644 --- a/examples/tlbr/tlbr_batch.sh +++ b/examples/tlbr/tlbr_batch.sh @@ -1,11 +1,16 @@ #!/bin/bash -# Example batch script for running tlbr example on a SLURM cluster +# Example batch script for running the tlbr example on a SLURM cluster -# set the number of nodesT -#SBATCH --nodes=4 +# The processors reserved here are sized to the fit in tlbr.conf: differential +# evolution with population_size=250 and smoothing=3 runs 250 x 3 = 750 simulations +# per iteration, which 5 nodes of 30 processors run in exactly 5 full waves. See +# "Sizing a run" in the PyBNF documentation before changing either number. -# set the number of cpus per node. -#SBATCH --mincpus=32 +# set the number of nodes +#SBATCH --nodes=5 + +# set the number of cpus per node +#SBATCH --mincpus=30 # set max wallclock time for the entire fitting job (1 day) #SBATCH --time=1-00:00:00 @@ -13,9 +18,12 @@ # set name of job #SBATCH --job-name=pybnf -# Enable Anaconda Python 3.5 -# Your cluster might require something different here, or might not require anything. -module load anaconda/Anaconda3 +# EDIT THIS LINE for your cluster: load a module (or activate a virtual environment) +# that provides Python 3.11 or newer with PyBNF installed. Some clusters need nothing +# here at all. +module load python/3.11 -# Run PyBNF -pybnf -c tlbr.conf -t SLURM -o +# Run PyBNF. Use "-t slurm-srun" instead if your cluster's nodes cannot be logged +# into over SSH with a key or a password; see "Which ways of starting a run log in to +# other machines" in the PyBNF documentation. +pybnf -c tlbr.conf -t slurm -o