diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index d39d029d..cf4ede86 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -40,3 +40,45 @@ The `bngsim` dependency is available from the package index and is resolved by The project uses a populated `uv.lock`, so normal `uv` dependency resolution and synchronization should remain enabled. + +## Testing multi-machine functionality + +PyBNF includes a full test suite in `tests/full_tests/` that validates +multi-machine cluster execution. If you're making changes to cluster +communication, distributed execution, or Dask integration, you should run these +tests. + +### Quick local test + +Run all tests on a single machine (no cluster required): + +```sh +cd tests/full_tests +python3 run_all.py +``` + +This takes about 30 minutes and writes results to `test_summary.txt`. + +### Cluster testing + +If you have access to a SLURM cluster, you can test multi-machine execution: + +```sh +cd tests/full_tests +# Edit the Python environment activation line in the script +sbatch cluster.sh # SSH-based automatic setup +# or +sbatch cluster_manual.sh # Manual Dask cluster setup +``` + +See `tests/full_tests/README.md` for detailed instructions on: +- Configuring the scripts for your cluster +- Adjusting resource allocation +- Interpreting test results +- Troubleshooting cluster issues + +**When to run cluster tests:** +- Changes to `pybnf/cluster.py` +- Changes to SSH or Dask worker management +- Changes to distributed algorithm execution +- Before releases (strongly recommended) diff --git a/tests/full_tests/README.md b/tests/full_tests/README.md new file mode 100644 index 00000000..e1cf511e --- /dev/null +++ b/tests/full_tests/README.md @@ -0,0 +1,140 @@ +# PyBNF Full Test Suite + +This directory contains a comprehensive test suite for validating PyBNF functionality, including multi-machine cluster execution. + +## Test Cases + +The suite includes 7 test problems (T1-T7) that exercise different PyBNF features: +- **T1-ssprop**: Scatter search with polynomial fitting +- **T2-ade-abcd**: Asynchronous differential evolution with refinement +- **T3-de-egg**: Differential evolution with bootstrap +- **T4-pso-nf**: Particle swarm optimization +- **T5-pt-trivial**: Parallel tempering on trivial problem +- **T6-check**: Model checking (`fit_type = check`) +- **T7-dream-trivial**: DREAM algorithm on trivial problem + +## Running the Tests + +### Local Execution (Single Machine) + +The simplest way to run the test suite: + +```bash +python3 run_all.py +``` + +This runs all tests locally without requiring cluster configuration. Results are written to `test_summary.txt`. + +**Expected runtime:** ~30 minutes on a typical workstation + +### Cluster Execution (Multi-Machine) + +Two SLURM batch scripts are provided for running on a cluster: + +#### Option 1: Automatic Cluster Setup (`cluster.sh`) + +Uses PyBNF's built-in SSH-based cluster management: + +```bash +sbatch cluster.sh +``` + +**Before running:** +1. Edit the `PYTHON ENVIRONMENT` section to activate your PyBNF environment +2. Adjust resource configuration if needed (nodes, CPUs, time limit) + +#### Option 2: Manual Dask Cluster (`cluster_manual.sh`) + +Manually configures a Dask scheduler and workers: + +```bash +sbatch cluster_manual.sh +``` + +**Before running:** +1. Edit the `PYTHON ENVIRONMENT` section to activate your PyBNF environment +2. Edit `WORKERS_PER_NODE` and `THREADS_PER_WORKER` to match your resources +3. Adjust resource configuration if needed (nodes, CPUs, time limit) + +**When to use this:** +- When you need fine-grained control over Dask worker configuration +- When automatic SSH setup doesn't work on your cluster +- When debugging cluster connectivity issues + +### Resource Requirements + +**Default configuration:** +- **Nodes:** 2 +- **CPUs per node:** 36 +- **Time limit:** 1 hour +- **Total cores:** 72 + +These values can be modified at the top of each batch script. + +## Interpreting Results + +### Output Files + +After running the test suite, check: + +- **Summary file**: `run_all.py` writes `test_summary.txt` for a local run, + `test_summary_ssh.txt` for the `ssh` mode, and `test_summary_sf.txt` for the + `sf` mode. +- **`T*/fit/Results/`**: Detailed results for each test case + +### Reference Output + +Example outputs are provided for comparison: + +- **`example_summary.txt`**: Single-machine run from March 2019 (v1.0.0 release) +- **`example_summary_ssh.txt`**: Multi-machine run using SSH mode +- **`example_summary_sf.txt`**: Multi-machine run using manual dask setup + +**Note:** These reference files are historical. Your output will differ due to: +- Algorithm changes and improvements +- Different random number seeds +- Hardware differences (CPU speed, core count) +- Numerical precision variations + +**What to check:** +- All tests should complete without errors +- Objective function values should be reasonable (similar order of magnitude) +- Test problems should converge (not diverge or fail) + +## Troubleshooting + +### Environment Issues + +If you see `ModuleNotFoundError` or import errors: +- Check that the active environment has PyBNF installed. Run `uv pip list | grep pybnf`, or `pip list | grep pybnf` if you installed PyBNF with pip. +- Install the dependencies with `uv sync`. This project is managed with uv and a populated `uv.lock` file, and it has no `requirements.txt`. See `CONTRIBUTING.md` for more. +- Make sure the batch script activates the correct environment. + +### Cluster Issues + +If workers fail to connect: +- Check SSH connectivity between nodes: `ssh hostname` +- Verify the scheduler file (`sf`) is on a shared filesystem +- Check firewall settings allow communication on Dask ports (8786, 8787) +- Review the SLURM job output file for error messages + +### Dask Command Not Found + +If you see `dask: command not found`: +- Check that the distributed package is installed. Run `uv pip list | grep distributed`. +- Make sure the environment is activated. +- This project needs the distributed package at version 2024.1.0 or newer, as set in `pyproject.toml`. The cluster commands are `dask scheduler` and `dask worker`, which are subcommands of the `dask` program. The older standalone programs named `dask-scheduler` and `dask-worker` stopped installing in distributed version 2026.6.0, as noted in `CHANGELOG.md`. + +## Contributing + +When modifying the multi-machine execution code in PyBNF: +1. Run the local test suite: `python3 run_all.py` +2. If you have cluster access, run one of the cluster tests +3. Compare results to verify no regressions +4. Document any new configuration requirements + +## See Also + +- [PyBNF Documentation](https://pybnf.readthedocs.io/) +- [Dask Distributed CLI Documentation](https://docs.dask.org/en/stable/deploying-cli.html) +- [CONTRIBUTING.md](../../CONTRIBUTING.md) - General contribution guidelines diff --git a/tests/full_tests/cluster.sh b/tests/full_tests/cluster.sh index e39e0396..e56b5252 100644 --- a/tests/full_tests/cluster.sh +++ b/tests/full_tests/cluster.sh @@ -1,23 +1,44 @@ #!/bin/bash -# set the number of nodes +#============================================================================= +# RESOURCE CONFIGURATION - Edit these values for your cluster +#============================================================================= +# Number of nodes to use #SBATCH --nodes=2 +# Minimum CPUs per node #SBATCH --mincpus=36 -# set max wallclock time for the entire fitting job +# Maximum wallclock time for the job #SBATCH --time=1:00:00 -# set name of job -#SBATCH --job-name=tests +# Job name +#SBATCH --job-name=pybnf-tests #SBATCH --exclusive - -# Enable custom Python 3.7.1 -# Your cluster might require something different here, or might not require anything. -source $HOME/rattlesnake/diamondback/bin/activate +#============================================================================= +# PYTHON ENVIRONMENT - Edit this line to activate your Python environment +#============================================================================= +# Uncomment and edit one of these lines, or add your own: +# source /path/to/your/virtualenv/bin/activate +# conda activate your-env-name +# module load python/3.11 +# +# Example (edit the path): +# source $HOME/path/to/pybnf-env/bin/activate + +# REQUIRED: Activate your Python environment here +# This environment must have PyBNF and its dependencies installed +source /path/to/your/pybnf-env/bin/activate # EDIT THIS LINE + +#============================================================================= +# SYSTEM LIMITS (optional) +#============================================================================= ulimit -u 500000 -# Run the test script +#============================================================================= +# RUN THE TEST SUITE +#============================================================================= +# Uses PyBNF's automatic SSH-based cluster setup python3 run_all.py ssh diff --git a/tests/full_tests/cluster_manual.sh b/tests/full_tests/cluster_manual.sh index c2bc1c9d..5a3cc3fb 100644 --- a/tests/full_tests/cluster_manual.sh +++ b/tests/full_tests/cluster_manual.sh @@ -1,30 +1,62 @@ #!/bin/bash -# set the number of nodes +#============================================================================= +# RESOURCE CONFIGURATION - Edit these values for your cluster +#============================================================================= +# Number of nodes to use #SBATCH --nodes=2 +# Minimum CPUs per node #SBATCH --mincpus=36 -# set max wallclock time for the entire fitting job +# Maximum wallclock time for the job #SBATCH --time=1:00:00 -# set name of job -#SBATCH --job-name=tests +# Job name +#SBATCH --job-name=pybnf-tests #SBATCH --exclusive -# Enable Python virtual environment. Edit this depending on your Python configuration. -source $P/python_envs/env1/bin/activate - +#============================================================================= +# WORKER CONFIGURATION - Edit these values to match your resources +#============================================================================= +# Number of workers per node (should match --mincpus above) +WORKERS_PER_NODE=36 + +# Number of threads per worker +THREADS_PER_WORKER=1 + +#============================================================================= +# PYTHON ENVIRONMENT - Edit this line to activate your Python environment +#============================================================================= +# Uncomment and edit one of these lines, or add your own: +# source /path/to/your/virtualenv/bin/activate +# conda activate your-env-name +# module load python/3.11 +# +# Example (edit the path): +# source $HOME/path/to/pybnf-env/bin/activate + +# REQUIRED: Activate your Python environment here +# This environment must have PyBNF and its dependencies installed +source /path/to/your/pybnf-env/bin/activate # EDIT THIS LINE + +#============================================================================= +# DASK CLUSTER SETUP +#============================================================================= # Automatically set up the dask scheduler and workers on the cluster allocation. -# This block should probably work for any SLURM cluster. +# This block should work for most SLURM clusters. # `dask scheduler` and `dask worker` are subcommands of the single `dask` program; the # separate dask-scheduler and dask-worker programs were dropped in distributed 2026.6.0. dask scheduler --scheduler-file sf & daskpath=$(which dask) scontrol show hostname $SLURM_JOB_NODELIST | while read node; do - ssh -n -f $node "cd $PWD ; nohup $daskpath worker --scheduler-file sf --nthreads 1 --nworkers 36 > /dev/null 2>&1 &" + ssh -n -f $node "cd $PWD ; nohup $daskpath worker --scheduler-file sf --nthreads $THREADS_PER_WORKER --nworkers $WORKERS_PER_NODE > /dev/null 2>&1 &" done -# Run the test script +#============================================================================= +# RUN THE TEST SUITE +#============================================================================= +# Uses the manually configured dask cluster (scheduler file: sf) python3 run_all.py sf +