added changes and notes from testing

monitoring-and-debugging.md

@@ -108,12 +108,19 @@ If `nsys` isn't found, refuses to run, or warns that sampling is restricted, see
 
 The rest of this section works through an xarray + CuPy example on real climate data, so the environment needs those libraries (xarray, netCDF4, CuPy, and cupy-xarray) in addition to the profilers.
 
-Use the Python environment you created earlier in the tutorial. The exact package manager doesn't matter. What matters is that `python` points to the environment you need.
+Use the Python environment you created earlier in the tutorial. The exact package manager doesn't matter. What matters is that `python` points to the environment you need. Activate it first. On Brev, the uv path installed into the system venv at `~/.venv`:
+
+```bash
+source ~/.venv/bin/activate
+```
+
+(If you used conda, run `conda activate tutorial-env` instead and if you used pixi, use `pixi shell --manifest-path envs/pixi.toml`)
+
+Then confirm `python` points where you expect:
 
 ```bash
 which python
 python --version
-python -m pip --version
 ```
 
 Verify that you have all the packages needed for this section:
@@ -214,8 +221,6 @@ Even with `synchronize()`, treat these timings as good estimates, not production
 
 The CPU and GPU have separate memory. A NumPy array lives in host memory. A CuPy array lives in device memory. Moving data from host to device is a CPU-to-GPU transfer, and moving it back is a GPU-to-CPU transfer.
 
-<!-- TODO(image): add a host/device memory diagram here, separate system RAM vs VRAM joined by a narrow PCIe link, with labeled H2D (cp.asarray) and D2H (.get()) arrows, to show why transfers cost. Pull a suitable diagram from the CUDA repo. Tracking issue: TBD. -->
-
 Those transfers aren't free. For a small workload, the transfer overhead can outweigh the benefit of GPU computation. For a larger workflow, repeated transfers can erase an otherwise good speedup. This is why just using the GPU isn't enough. The useful question is whether enough of the expensive work stayed on the GPU long enough to justify the transfer.
 
 Some rules of thumb to use:
@@ -316,7 +321,7 @@ Compared with the CPU baseline, this port moves the field onto the GPU for the r
  U, S, Vt = np.linalg.svd(X, full_matrices=False)
 ```
 
-> **Exercise:** before running it, predict the timing. This script does the data load, reshape, and mean-subtraction on the GPU, then runs the SVD. Will it beat the CPU baseline? Run it, then profile it the same way you profiled the baseline (`python -m cProfile -o profile-xarray-cupy-naive.prof scripts/xarray-cupy-naive.py`, then `python -m snakeviz profile-xarray-cupy-naive.prof`) and see which call still dominates the runtime.
+> **Exercise:** before running it, predict the timing. This script does the data load, reshape, and mean-subtraction on the GPU, then runs the SVD. Will it beat the CPU baseline? Run it, then profile it the same way you profiled the baseline (`python -m cProfile -o profile-xarray-cupy-naive.prof scripts/xarray-cupy-naive.py`, then `python -m snakeviz -s profile-xarray-cupy-naive.prof`) and see which call still dominates the runtime.
 
 The mistake lives in these two lines:
 
@@ -345,16 +350,45 @@ This is the moment `nvtop` is the right tool. You'll see the GPU light up briefl
 
 ```bash
 nsys profile \
-  --trace cuda,osrt,nvtx \
+  --trace cuda,osrt \
   --cuda-memory-usage true \
   --force-overwrite true \
   --output profile-xarray-cupy-naive \
   $(which python) scripts/xarray-cupy-naive.py
 ```
 
+These are what the various flags are for (For more information, check the [Nsight User Guide](https://docs.nvidia.com/nsight-systems/UserGuide/index.html#cli-profile-command-switch-options)):
+
+- `--trace cuda,osrt` : record GPU/CUDA activity plus CPU-side OS-runtime waits.
+- `--cuda-memory-usage true` : track GPU memory allocation over the run.
+- `--force-overwrite true` : replace an existing report of the same name.
+- `--output profile-xarray-cupy-naive` : name of the `.nsys-rep` file.
+- `$(which python) scripts/xarray-cupy-naive.py` : the program to profile
+
+At the end of the run, `nsys` prints where it wrote the report. On Brev that's your home directory:
+
+```text
+Generated:
+    /home/ubuntu/profile-xarray-cupy-naive.nsys-rep
+```
+
+Capturing happens on the VM, but exploring the timeline needs the **Nsight Systems desktop app**, which runs on your laptop, not the VM. So you pull the report down and open it locally.
+
+Install the Nsight Systems GUI on your laptop once, from the [Nsight Systems page](https://developer.nvidia.com/nsight-systems). This desktop application is separate from the `nsys` command-line tool you installed on the VM.
+
+Then copy the report down from Brev. Run this from a new terminal on your laptop (not the VM) in the location you want the file to be saved:
+
+```bash
+brev copy <your-instance-name>:~/profile-xarray-cupy-naive.nsys-rep .
+```
+
+`brev copy` takes `<source> <destination>`, so a `<instance>:<remote-path>` source and a local `.` destination pulls the `.nsys-rep` into your current directory. If `~` doesn't resolve on the remote, use the absolute path `/home/ubuntu/profile-xarray-cupy-naive.nsys-rep`.
+
+Finally, open it: launch Nsight Systems and use **File > Open** to select the `.nsys-rep` (or run `nsys-ui profile-xarray-cupy-naive.nsys-rep` from a terminal if the GUI's launcher is on your `PATH`). Zoom into the `CUDA HW` rows to explore the timeline.
+
 ![Nsight Systems timeline of xarray-cupy-naive.py](images/nsight-xarray-cupy-naive.png)
 
-These timelines are dense, so here's how to read them. Time runs left to right. The rows under `CUDA HW` show what the GPU actually did: the `Kernels` row is GPU compute, and the `Memory` row is the host-to-device and device-to-host copies. When those rows are empty, the GPU is idle. You produce the `.nsys-rep` on the VM, but exploring the timeline needs the Nsight Systems desktop app, so download the file to your laptop and open it there, or just follow the annotated screenshots in this section.
+These timelines are dense, so here's how to read them. Time runs left to right. The rows under `CUDA HW` show what the GPU actually did: the `Kernels` row is GPU compute, and the `Memory` row is the host-to-device and device-to-host copies. When those rows are empty, the GPU is idle. If you are having trouble opening the report, follow the annotated screenshots in this section.
 
 Reading the naive run left to right: a host-to-device `cudaMemcpy` near the start (the `cp.asarray(air)` call), a few small kernels for the reshape and mean-subtraction, then a long device-to-host `cudaMemcpy` (the `.get()`), and after that a long stretch where the `Kernels` and `Memory` rows are empty while the CPU runs SVD. That idle stretch is the bottleneck.
 
@@ -392,13 +426,19 @@ python scripts/xarray-cupy-fixed.py
 
 ```bash
 nsys profile \
-  --trace cuda,osrt,nvtx \
+  --trace cuda,osrt \
   --cuda-memory-usage true \
   --force-overwrite true \
   --output profile-xarray-cupy-fixed \
   $(which python) scripts/xarray-cupy-fixed.py
 ```
 
+Copy this report down to your laptop and open it in Nsight Systems, the same way you did for the naive run:
+
+```bash
+brev copy <your-instance-name>:~/profile-xarray-cupy-fixed.nsys-rep .
+```
+
 ![Nsight Systems timeline of xarray-cupy-fixed.py](images/nsight-xarray-cupy-fixed.png)
 
 Compared with the naive version, the timeline shows no host-to-device or device-to-host memcpy in the middle of the run, and the long idle stretch is gone. The `Kernels` row stays busy straight through the SVD. That's the win.
@@ -477,13 +517,19 @@ python scripts/preprocess-gpu-loop.py
 
 ```bash
 nsys profile \
-  --trace cuda,osrt,nvtx \
+  --trace cuda,osrt \
   --cuda-memory-usage true \
   --force-overwrite true \
   --output profile-preprocess-gpu-loop \
   $(which python) scripts/preprocess-gpu-loop.py
 ```
 
+Copy this report down and open it in Nsight Systems as before:
+
+```bash
+brev copy <your-instance-name>:~/profile-preprocess-gpu-loop.nsys-rep .
+```
+
 ![Nsight Systems timeline of the per-cell GPU loop](images/nsight-preprocess-gpu-loop.png)
 
 One viewing note for this run: at full zoom it looks like a single solid bar, because there are roughly 150,000 launches packed together. Zoom into a window of a few tens of microseconds and the individual tiny kernels (`cupy_mean`, `cupy_var`, and friends) separate out, each followed by a gap. `cProfile` and SnakeViz would flag the symptom here which is thousands of tiny CuPy calls with a high call count, but they measure Python-side time and can't show that the GPU sits idle between those launches. The timeline can: a tightly packed row of very short kernel launches with thin gaps between them. Each cell triggers several tiny kernels, and at thousands of cells the launch overhead dominates the actual arithmetic. The GPU spends more time waiting between launches than computing. Using the GPU naively, one tiny operation at a time, is slower than the CPU loop, not faster.