From 26bf00a77595778466d82f186559505323ba83ad Mon Sep 17 00:00:00 2001 From: Jaya Venkatesh Date: Fri, 10 Jul 2026 12:47:05 -0700 Subject: [PATCH 1/4] added changes and notes from testing --- monitoring-and-debugging.md | 54 ++++++++++++++++++++++++++++++++----- 1 file changed, 47 insertions(+), 7 deletions(-) diff --git a/monitoring-and-debugging.md b/monitoring-and-debugging.md index daa1914..ecb6a0b 100644 --- a/monitoring-and-debugging.md +++ b/monitoring-and-debugging.md @@ -108,12 +108,19 @@ If `nsys` isn't found, refuses to run, or warns that sampling is restricted, see The rest of this section works through an xarray + CuPy example on real climate data, so the environment needs those libraries (xarray, netCDF4, CuPy, and cupy-xarray) in addition to the profilers. -Use the Python environment you created earlier in the tutorial. The exact package manager doesn't matter. What matters is that `python` points to the environment you need. +Use the Python environment you created earlier in the tutorial. The exact package manager doesn't matter. What matters is that `python` points to the environment you need. Activate it first. On Brev, the uv path installed into the system venv at `~/.venv`: + +```bash +source ~/.venv/bin/activate +``` + +(If you used conda, run `conda activate tutorial-env` instead; if pixi, run the section's commands through `pixi run` from `envs/`, or open a `pixi shell`.) + +Then confirm `python` points where you expect: ```bash which python python --version -python -m pip --version ``` Verify that you have all the packages needed for this section: @@ -316,7 +323,7 @@ Compared with the CPU baseline, this port moves the field onto the GPU for the r U, S, Vt = np.linalg.svd(X, full_matrices=False) ``` -> **Exercise:** before running it, predict the timing. This script does the data load, reshape, and mean-subtraction on the GPU, then runs the SVD. Will it beat the CPU baseline? Run it, then profile it the same way you profiled the baseline (`python -m cProfile -o profile-xarray-cupy-naive.prof scripts/xarray-cupy-naive.py`, then `python -m snakeviz profile-xarray-cupy-naive.prof`) and see which call still dominates the runtime. +> **Exercise:** before running it, predict the timing. This script does the data load, reshape, and mean-subtraction on the GPU, then runs the SVD. Will it beat the CPU baseline? Run it, then profile it the same way you profiled the baseline (`python -m cProfile -o profile-xarray-cupy-naive.prof scripts/xarray-cupy-naive.py`, then `python -m snakeviz -s profile-xarray-cupy-naive.prof`) and see which call still dominates the runtime. The mistake lives in these two lines: @@ -345,16 +352,37 @@ This is the moment `nvtop` is the right tool. You'll see the GPU light up briefl ```bash nsys profile \ - --trace cuda,osrt,nvtx \ + --trace cuda,osrt \ --cuda-memory-usage true \ --force-overwrite true \ --output profile-xarray-cupy-naive \ $(which python) scripts/xarray-cupy-naive.py ``` +At the end of the run, `nsys` prints where it wrote the report. On Brev that's your home directory: + +```text +Generated: + /home/ubuntu/profile-xarray-cupy-naive.nsys-rep +``` + +Capturing happens on the VM, but exploring the timeline needs the **Nsight Systems desktop app**, which runs on your laptop, not the VM. So you pull the report down and open it locally. + +Install the Nsight Systems GUI on your laptop once, from the [Nsight Systems page](https://developer.nvidia.com/nsight-systems). This desktop application is separate from the `nsys` command-line tool you installed on the VM. + +Then copy the report down from Brev. Run this from a new terminal on your laptop (not the VM): + +```bash +brev copy :~/profile-xarray-cupy-naive.nsys-rep . +``` + +`brev copy` takes ` `, so a `:` source and a local `.` destination pulls the `.nsys-rep` into your current directory. If `~` doesn't resolve on the remote, use the absolute path `/home/ubuntu/profile-xarray-cupy-naive.nsys-rep`. + +Finally, open it: launch Nsight Systems and use **File > Open** to select the `.nsys-rep` (or run `nsys-ui profile-xarray-cupy-naive.nsys-rep` from a terminal if the GUI's launcher is on your `PATH`). Zoom into the `CUDA HW` rows to explore the timeline. + ![Nsight Systems timeline of xarray-cupy-naive.py](images/nsight-xarray-cupy-naive.png) -These timelines are dense, so here's how to read them. Time runs left to right. The rows under `CUDA HW` show what the GPU actually did: the `Kernels` row is GPU compute, and the `Memory` row is the host-to-device and device-to-host copies. When those rows are empty, the GPU is idle. You produce the `.nsys-rep` on the VM, but exploring the timeline needs the Nsight Systems desktop app, so download the file to your laptop and open it there, or just follow the annotated screenshots in this section. +These timelines are dense, so here's how to read them. Time runs left to right. The rows under `CUDA HW` show what the GPU actually did: the `Kernels` row is GPU compute, and the `Memory` row is the host-to-device and device-to-host copies. When those rows are empty, the GPU is idle. If you are having trouble opening the report, follow the annotated screenshots in this section. Reading the naive run left to right: a host-to-device `cudaMemcpy` near the start (the `cp.asarray(air)` call), a few small kernels for the reshape and mean-subtraction, then a long device-to-host `cudaMemcpy` (the `.get()`), and after that a long stretch where the `Kernels` and `Memory` rows are empty while the CPU runs SVD. That idle stretch is the bottleneck. @@ -392,13 +420,19 @@ python scripts/xarray-cupy-fixed.py ```bash nsys profile \ - --trace cuda,osrt,nvtx \ + --trace cuda,osrt \ --cuda-memory-usage true \ --force-overwrite true \ --output profile-xarray-cupy-fixed \ $(which python) scripts/xarray-cupy-fixed.py ``` +Copy this report down to your laptop and open it in Nsight Systems, the same way you did for the naive run: + +```bash +brev copy :~/profile-xarray-cupy-fixed.nsys-rep . +``` + ![Nsight Systems timeline of xarray-cupy-fixed.py](images/nsight-xarray-cupy-fixed.png) Compared with the naive version, the timeline shows no host-to-device or device-to-host memcpy in the middle of the run, and the long idle stretch is gone. The `Kernels` row stays busy straight through the SVD. That's the win. @@ -477,13 +511,19 @@ python scripts/preprocess-gpu-loop.py ```bash nsys profile \ - --trace cuda,osrt,nvtx \ + --trace cuda,osrt \ --cuda-memory-usage true \ --force-overwrite true \ --output profile-preprocess-gpu-loop \ $(which python) scripts/preprocess-gpu-loop.py ``` +Copy this report down and open it in Nsight Systems as before: + +```bash +brev copy :~/profile-preprocess-gpu-loop.nsys-rep . +``` + ![Nsight Systems timeline of the per-cell GPU loop](images/nsight-preprocess-gpu-loop.png) One viewing note for this run: at full zoom it looks like a single solid bar, because there are roughly 150,000 launches packed together. Zoom into a window of a few tens of microseconds and the individual tiny kernels (`cupy_mean`, `cupy_var`, and friends) separate out, each followed by a gap. `cProfile` and SnakeViz would flag the symptom here which is thousands of tiny CuPy calls with a high call count, but they measure Python-side time and can't show that the GPU sits idle between those launches. The timeline can: a tightly packed row of very short kernel launches with thin gaps between them. Each cell triggers several tiny kernels, and at thousands of cells the launch overhead dominates the actual arithmetic. The GPU spends more time waiting between launches than computing. Using the GPU naively, one tiny operation at a time, is slower than the CPU loop, not faster. From d92cc8e0ad23228e98336bead0a230d5a0a88ccd Mon Sep 17 00:00:00 2001 From: Jaya Venkatesh Date: Fri, 10 Jul 2026 14:03:36 -0700 Subject: [PATCH 2/4] Update monitoring-and-debugging.md Co-authored-by: Naty Clementi --- monitoring-and-debugging.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/monitoring-and-debugging.md b/monitoring-and-debugging.md index ecb6a0b..ca2ad5d 100644 --- a/monitoring-and-debugging.md +++ b/monitoring-and-debugging.md @@ -370,7 +370,7 @@ Capturing happens on the VM, but exploring the timeline needs the **Nsight Syste Install the Nsight Systems GUI on your laptop once, from the [Nsight Systems page](https://developer.nvidia.com/nsight-systems). This desktop application is separate from the `nsys` command-line tool you installed on the VM. -Then copy the report down from Brev. Run this from a new terminal on your laptop (not the VM): +Then copy the report down from Brev. Run this from a new terminal on your laptop (not the VM) in the location you want the file to be saved: ```bash brev copy :~/profile-xarray-cupy-naive.nsys-rep . From d40257b9e9db2589baf4113d471cc2457e9921aa Mon Sep 17 00:00:00 2001 From: Jaya Venkatesh Date: Fri, 10 Jul 2026 14:12:35 -0700 Subject: [PATCH 3/4] add pixi command and nsight flags explanations --- monitoring-and-debugging.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/monitoring-and-debugging.md b/monitoring-and-debugging.md index ca2ad5d..e62e4b9 100644 --- a/monitoring-and-debugging.md +++ b/monitoring-and-debugging.md @@ -114,7 +114,7 @@ Use the Python environment you created earlier in the tutorial. The exact packag source ~/.venv/bin/activate ``` -(If you used conda, run `conda activate tutorial-env` instead; if pixi, run the section's commands through `pixi run` from `envs/`, or open a `pixi shell`.) +(If you used conda, run `conda activate tutorial-env` instead and if you used pixi, use `pixi shell --manifest-path envs/pixi.toml`) Then confirm `python` points where you expect: @@ -221,8 +221,6 @@ Even with `synchronize()`, treat these timings as good estimates, not production The CPU and GPU have separate memory. A NumPy array lives in host memory. A CuPy array lives in device memory. Moving data from host to device is a CPU-to-GPU transfer, and moving it back is a GPU-to-CPU transfer. - - Those transfers aren't free. For a small workload, the transfer overhead can outweigh the benefit of GPU computation. For a larger workflow, repeated transfers can erase an otherwise good speedup. This is why just using the GPU isn't enough. The useful question is whether enough of the expensive work stayed on the GPU long enough to justify the transfer. Some rules of thumb to use: @@ -359,6 +357,14 @@ nsys profile \ $(which python) scripts/xarray-cupy-naive.py ``` +These are what the various flags are for: + +- `--trace cuda,osrt` : record GPU/CUDA activity plus CPU-side OS-runtime waits. +- `--cuda-memory-usage true` : track GPU memory allocation over the run. +- `--force-overwrite true` : replace an existing report of the same name. +- `--output profile-xarray-cupy-naive` : name of the `.nsys-rep` file. +- `$(which python) scripts/xarray-cupy-naive.py` : the program to profile + At the end of the run, `nsys` prints where it wrote the report. On Brev that's your home directory: ```text From aeee8494761142f5a55a5bcef522ac322cb021a2 Mon Sep 17 00:00:00 2001 From: Jaya Venkatesh Date: Fri, 10 Jul 2026 14:20:06 -0700 Subject: [PATCH 4/4] add link to Nsight documentation --- monitoring-and-debugging.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/monitoring-and-debugging.md b/monitoring-and-debugging.md index e62e4b9..ac504e7 100644 --- a/monitoring-and-debugging.md +++ b/monitoring-and-debugging.md @@ -357,7 +357,7 @@ nsys profile \ $(which python) scripts/xarray-cupy-naive.py ``` -These are what the various flags are for: +These are what the various flags are for (For more information, check the [Nsight User Guide](https://docs.nvidia.com/nsight-systems/UserGuide/index.html#cli-profile-command-switch-options)): - `--trace cuda,osrt` : record GPU/CUDA activity plus CPU-side OS-runtime waits. - `--cuda-memory-usage true` : track GPU memory allocation over the run.