From fe3c4ccb4f279d1776f29f04606ac115d716a2a1 Mon Sep 17 00:00:00 2001 From: psamuelvijay Date: Sat, 22 Aug 2026 10:56:57 +0530 Subject: [PATCH 1/2] Improve multi-machine test script configuration and documentation (#624) - Replace hardcoded Python environment paths with clear edit instructions - Make resource configuration (nodes, CPUs, time) easy to find and modify - Extract hardcoded worker counts to variables in cluster_manual.sh - Add comprehensive README.md in tests/full_tests/ explaining: - How to run tests locally vs on cluster - How to configure each script - Resource requirements - How to interpret results and reference output - Troubleshooting common issues - Update CONTRIBUTING.md with multi-machine testing guidelines - Document when cluster tests should be run The maintainer already fixed dask command names (dask-scheduler -> dask scheduler) in a recent commit. This PR addresses the remaining configuration and documentation issues from the original issue. Signed-off-by: psamuelvijay --- CONTRIBUTING.md | 42 +++++++++ tests/full_tests/README.md | 138 +++++++++++++++++++++++++++++ tests/full_tests/cluster.sh | 39 ++++++-- tests/full_tests/cluster_manual.sh | 52 ++++++++--- 4 files changed, 252 insertions(+), 19 deletions(-) create mode 100644 tests/full_tests/README.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index d39d029d2..cf4ede865 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -40,3 +40,45 @@ The `bngsim` dependency is available from the package index and is resolved by The project uses a populated `uv.lock`, so normal `uv` dependency resolution and synchronization should remain enabled. + +## Testing multi-machine functionality + +PyBNF includes a full test suite in `tests/full_tests/` that validates +multi-machine cluster execution. If you're making changes to cluster +communication, distributed execution, or Dask integration, you should run these +tests. + +### Quick local test + +Run all tests on a single machine (no cluster required): + +```sh +cd tests/full_tests +python3 run_all.py +``` + +This takes about 30 minutes and writes results to `test_summary.txt`. + +### Cluster testing + +If you have access to a SLURM cluster, you can test multi-machine execution: + +```sh +cd tests/full_tests +# Edit the Python environment activation line in the script +sbatch cluster.sh # SSH-based automatic setup +# or +sbatch cluster_manual.sh # Manual Dask cluster setup +``` + +See `tests/full_tests/README.md` for detailed instructions on: +- Configuring the scripts for your cluster +- Adjusting resource allocation +- Interpreting test results +- Troubleshooting cluster issues + +**When to run cluster tests:** +- Changes to `pybnf/cluster.py` +- Changes to SSH or Dask worker management +- Changes to distributed algorithm execution +- Before releases (strongly recommended) diff --git a/tests/full_tests/README.md b/tests/full_tests/README.md new file mode 100644 index 000000000..c7de59099 --- /dev/null +++ b/tests/full_tests/README.md @@ -0,0 +1,138 @@ +# PyBNF Full Test Suite + +This directory contains a comprehensive test suite for validating PyBNF functionality, including multi-machine cluster execution. + +## Test Cases + +The suite includes 7 test problems (T1-T7) that exercise different PyBNF features: +- **T1-ssprop**: Scatter search with polynomial fitting +- **T2-ade-abcd**: Asynchronous differential evolution with refinement +- **T3-de-egg**: Differential evolution with bootstrap +- **T4-pso-nf**: Particle swarm optimization +- **T5-pt-trivial**: Parallel tempering on trivial problem +- **T6-check**: Configuration checking +- **T7-dream-trivial**: DREAM algorithm on trivial problem + +## Running the Tests + +### Local Execution (Single Machine) + +The simplest way to run the test suite: + +```bash +python3 run_all.py +``` + +This runs all tests locally without requiring cluster configuration. Results are written to `test_summary.txt`. + +**Expected runtime:** ~30 minutes on a typical workstation + +### Cluster Execution (Multi-Machine) + +Two SLURM batch scripts are provided for running on a cluster: + +#### Option 1: Automatic Cluster Setup (`cluster.sh`) + +Uses PyBNF's built-in SSH-based cluster management: + +```bash +sbatch cluster.sh +``` + +**Before running:** +1. Edit the `PYTHON ENVIRONMENT` section to activate your PyBNF environment +2. Adjust resource configuration if needed (nodes, CPUs, time limit) + +#### Option 2: Manual Dask Cluster (`cluster_manual.sh`) + +Manually configures a Dask scheduler and workers: + +```bash +sbatch cluster_manual.sh +``` + +**Before running:** +1. Edit the `PYTHON ENVIRONMENT` section to activate your PyBNF environment +2. Edit `WORKERS_PER_NODE` and `THREADS_PER_WORKER` to match your resources +3. Adjust resource configuration if needed (nodes, CPUs, time limit) + +**When to use this:** +- When you need fine-grained control over Dask worker configuration +- When automatic SSH setup doesn't work on your cluster +- When debugging cluster connectivity issues + +### Resource Requirements + +**Default configuration:** +- **Nodes:** 2 +- **CPUs per node:** 36 +- **Time limit:** 1 hour +- **Total cores:** 72 + +These values can be modified at the top of each batch script. + +## Interpreting Results + +### Output Files + +After running the test suite, check: + +- **`test_summary.txt`**: Summary of all test results +- **`T*/fit/Results/`**: Detailed results for each test case + +### Reference Output + +Example outputs are provided for comparison: + +- **`example_summary.txt`**: Single-machine run from March 2019 (v1.0.0 release) +- **`example_summary_ssh.txt`**: Multi-machine run using SSH mode +- **`example_summary_sf.txt`**: Multi-machine run using manual dask setup + +**Note:** These reference files are historical. Your output will differ due to: +- Algorithm changes and improvements +- Different random number seeds +- Hardware differences (CPU speed, core count) +- Numerical precision variations + +**What to check:** +- All tests should complete without errors +- Objective function values should be reasonable (similar order of magnitude) +- Test problems should converge (not diverge or fail) + +## Troubleshooting + +### Environment Issues + +If you see `ModuleNotFoundError` or import errors: +- Verify your Python environment has PyBNF installed: `pip list | grep pybnf` +- Check that all dependencies are installed: `pip install -r requirements.txt` +- Ensure you're activating the correct environment in the batch script + +### Cluster Issues + +If workers fail to connect: +- Check SSH connectivity between nodes: `ssh hostname` +- Verify the scheduler file (`sf`) is on a shared filesystem +- Check firewall settings allow communication on Dask ports (8786, 8787) +- Review the SLURM job output file for error messages + +### Dask Command Not Found + +If you see `dask: command not found`: +- Verify `distributed` package is installed: `pip list | grep distributed` +- Check that your environment is properly activated +- Ensure `distributed >= 2021.0.0` (the modern CLI was introduced in 2021) + +## Contributing + +When modifying the multi-machine execution code in PyBNF: +1. Run the local test suite: `python3 run_all.py` +2. If you have cluster access, run one of the cluster tests +3. Compare results to verify no regressions +4. Document any new configuration requirements + +## See Also + +- [PyBNF Documentation](https://pybnf.readthedocs.io/) +- [Dask Distributed CLI Documentation](https://docs.dask.org/en/stable/deploying-cli.html) +- [CONTRIBUTING.md](../../CONTRIBUTING.md) - General contribution guidelines diff --git a/tests/full_tests/cluster.sh b/tests/full_tests/cluster.sh index e39e0396f..4020d2993 100644 --- a/tests/full_tests/cluster.sh +++ b/tests/full_tests/cluster.sh @@ -1,23 +1,44 @@ #!/bin/bash -# set the number of nodes +#============================================================================= +# RESOURCE CONFIGURATION - Edit these values for your cluster +#============================================================================= +# Number of nodes to use #SBATCH --nodes=2 +# Minimum CPUs per node #SBATCH --mincpus=36 -# set max wallclock time for the entire fitting job +# Maximum wallclock time for the job #SBATCH --time=1:00:00 -# set name of job -#SBATCH --job-name=tests +# Job name +#SBATCH --job-name=pybnf-tests #SBATCH --exclusive - -# Enable custom Python 3.7.1 -# Your cluster might require something different here, or might not require anything. -source $HOME/rattlesnake/diamondback/bin/activate +#============================================================================= +# PYTHON ENVIRONMENT - Edit this line to activate your Python environment +#============================================================================= +# Uncomment and edit one of these lines, or add your own: +# source /path/to/your/virtualenv/bin/activate +# conda activate your-env-name +# module load python/3.11 +# +# Example (edit the path): +# source $HOME/path/to/pybnf-env/bin/activate + +# REQUIRED: Activate your Python environment here +# This environment must have PyBNF and its dependencies installed +source $HOME/rattlesnake/diamondback/bin/activate # EDIT THIS LINE + +#============================================================================= +# SYSTEM LIMITS (optional) +#============================================================================= ulimit -u 500000 -# Run the test script +#============================================================================= +# RUN THE TEST SUITE +#============================================================================= +# Uses PyBNF's automatic SSH-based cluster setup python3 run_all.py ssh diff --git a/tests/full_tests/cluster_manual.sh b/tests/full_tests/cluster_manual.sh index c2bc1c9d8..006e14708 100644 --- a/tests/full_tests/cluster_manual.sh +++ b/tests/full_tests/cluster_manual.sh @@ -1,30 +1,62 @@ #!/bin/bash -# set the number of nodes +#============================================================================= +# RESOURCE CONFIGURATION - Edit these values for your cluster +#============================================================================= +# Number of nodes to use #SBATCH --nodes=2 +# Minimum CPUs per node #SBATCH --mincpus=36 -# set max wallclock time for the entire fitting job +# Maximum wallclock time for the job #SBATCH --time=1:00:00 -# set name of job -#SBATCH --job-name=tests +# Job name +#SBATCH --job-name=pybnf-tests #SBATCH --exclusive -# Enable Python virtual environment. Edit this depending on your Python configuration. -source $P/python_envs/env1/bin/activate - +#============================================================================= +# WORKER CONFIGURATION - Edit these values to match your resources +#============================================================================= +# Number of workers per node (should match --mincpus above) +WORKERS_PER_NODE=36 + +# Number of threads per worker +THREADS_PER_WORKER=1 + +#============================================================================= +# PYTHON ENVIRONMENT - Edit this line to activate your Python environment +#============================================================================= +# Uncomment and edit one of these lines, or add your own: +# source /path/to/your/virtualenv/bin/activate +# conda activate your-env-name +# module load python/3.11 +# +# Example (edit the path): +# source $HOME/path/to/pybnf-env/bin/activate + +# REQUIRED: Activate your Python environment here +# This environment must have PyBNF and its dependencies installed +source $P/python_envs/env1/bin/activate # EDIT THIS LINE + +#============================================================================= +# DASK CLUSTER SETUP +#============================================================================= # Automatically set up the dask scheduler and workers on the cluster allocation. -# This block should probably work for any SLURM cluster. +# This block should work for most SLURM clusters. # `dask scheduler` and `dask worker` are subcommands of the single `dask` program; the # separate dask-scheduler and dask-worker programs were dropped in distributed 2026.6.0. dask scheduler --scheduler-file sf & daskpath=$(which dask) scontrol show hostname $SLURM_JOB_NODELIST | while read node; do - ssh -n -f $node "cd $PWD ; nohup $daskpath worker --scheduler-file sf --nthreads 1 --nworkers 36 > /dev/null 2>&1 &" + ssh -n -f $node "cd $PWD ; nohup $daskpath worker --scheduler-file sf --nthreads $THREADS_PER_WORKER --nworkers $WORKERS_PER_NODE > /dev/null 2>&1 &" done -# Run the test script +#============================================================================= +# RUN THE TEST SUITE +#============================================================================= +# Uses the manually configured dask cluster (scheduler file: sf) python3 run_all.py sf + From 4aab9bdf574a0f4332941e69ed878c47c39eff59 Mon Sep 17 00:00:00 2001 From: Bill Hlavacek Date: Sat, 22 Aug 2026 10:56:14 -0600 Subject: [PATCH 2/2] Correct factual mistakes in the full tests documentation While reviewing pull request 640 we found several statements in the new documentation that do not match this repository. This commit corrects them. The README described test T6 as configuration checking. T6 runs the model checking job type, which its configuration file selects with the setting fit_type = check. The README listed only test_summary.txt as the output file. The run_all.py script writes test_summary.txt for a local run, test_summary_ssh.txt for the ssh mode, and test_summary_sf.txt for the sf mode. The README told readers to install dependencies from a requirements.txt file. This project has no such file. It is managed with uv and a populated uv.lock file, so dependencies are installed with uv sync. The README said the distributed package must be version 2021.0.0 or newer. The project requires version 2024.1.0 or newer, as set in pyproject.toml. The older standalone programs named dask-scheduler and dask-worker stopped installing in distributed version 2026.6.0, as noted in CHANGELOG.md. The two cluster batch scripts still pointed at the original author's personal environment paths. This commit replaces them with a generic placeholder path so that running a script as written no longer tries to activate a real user's environment. --- tests/full_tests/README.md | 18 ++++++++++-------- tests/full_tests/cluster.sh | 2 +- tests/full_tests/cluster_manual.sh | 2 +- 3 files changed, 12 insertions(+), 10 deletions(-) diff --git a/tests/full_tests/README.md b/tests/full_tests/README.md index c7de59099..e1cf511ed 100644 --- a/tests/full_tests/README.md +++ b/tests/full_tests/README.md @@ -10,7 +10,7 @@ The suite includes 7 test problems (T1-T7) that exercise different PyBNF feature - **T3-de-egg**: Differential evolution with bootstrap - **T4-pso-nf**: Particle swarm optimization - **T5-pt-trivial**: Parallel tempering on trivial problem -- **T6-check**: Configuration checking +- **T6-check**: Model checking (`fit_type = check`) - **T7-dream-trivial**: DREAM algorithm on trivial problem ## Running the Tests @@ -77,7 +77,9 @@ These values can be modified at the top of each batch script. After running the test suite, check: -- **`test_summary.txt`**: Summary of all test results +- **Summary file**: `run_all.py` writes `test_summary.txt` for a local run, + `test_summary_ssh.txt` for the `ssh` mode, and `test_summary_sf.txt` for the + `sf` mode. - **`T*/fit/Results/`**: Detailed results for each test case ### Reference Output @@ -104,9 +106,9 @@ Example outputs are provided for comparison: ### Environment Issues If you see `ModuleNotFoundError` or import errors: -- Verify your Python environment has PyBNF installed: `pip list | grep pybnf` -- Check that all dependencies are installed: `pip install -r requirements.txt` -- Ensure you're activating the correct environment in the batch script +- Check that the active environment has PyBNF installed. Run `uv pip list | grep pybnf`, or `pip list | grep pybnf` if you installed PyBNF with pip. +- Install the dependencies with `uv sync`. This project is managed with uv and a populated `uv.lock` file, and it has no `requirements.txt`. See `CONTRIBUTING.md` for more. +- Make sure the batch script activates the correct environment. ### Cluster Issues @@ -119,9 +121,9 @@ If workers fail to connect: ### Dask Command Not Found If you see `dask: command not found`: -- Verify `distributed` package is installed: `pip list | grep distributed` -- Check that your environment is properly activated -- Ensure `distributed >= 2021.0.0` (the modern CLI was introduced in 2021) +- Check that the distributed package is installed. Run `uv pip list | grep distributed`. +- Make sure the environment is activated. +- This project needs the distributed package at version 2024.1.0 or newer, as set in `pyproject.toml`. The cluster commands are `dask scheduler` and `dask worker`, which are subcommands of the `dask` program. The older standalone programs named `dask-scheduler` and `dask-worker` stopped installing in distributed version 2026.6.0, as noted in `CHANGELOG.md`. ## Contributing diff --git a/tests/full_tests/cluster.sh b/tests/full_tests/cluster.sh index 4020d2993..e56b52520 100644 --- a/tests/full_tests/cluster.sh +++ b/tests/full_tests/cluster.sh @@ -30,7 +30,7 @@ # REQUIRED: Activate your Python environment here # This environment must have PyBNF and its dependencies installed -source $HOME/rattlesnake/diamondback/bin/activate # EDIT THIS LINE +source /path/to/your/pybnf-env/bin/activate # EDIT THIS LINE #============================================================================= # SYSTEM LIMITS (optional) diff --git a/tests/full_tests/cluster_manual.sh b/tests/full_tests/cluster_manual.sh index 006e14708..5a3cc3fbf 100644 --- a/tests/full_tests/cluster_manual.sh +++ b/tests/full_tests/cluster_manual.sh @@ -39,7 +39,7 @@ THREADS_PER_WORKER=1 # REQUIRED: Activate your Python environment here # This environment must have PyBNF and its dependencies installed -source $P/python_envs/env1/bin/activate # EDIT THIS LINE +source /path/to/your/pybnf-env/bin/activate # EDIT THIS LINE #============================================================================= # DASK CLUSTER SETUP