Handle gdrdrv kernel module in _unload_driver()

[shaq918]

Summary

The _unload_driver() function in the driver container entrypoint script accounts for nvidia_modeset, nvidia_uvm, and nvidia_peermem when computing nvidia_deps, but does not account for gdrdrv (loaded by the nvidia-gdrcopy-ctr sidecar container). This causes the driver container to enter CrashLoopBackOff when it restarts on nodes where GDRCopy is enabled.

Problem

When RDMA is enabled in the GPU Operator ClusterPolicy (driver.rdma.enabled: true), the driver pod runs three containers:

Container	Kernel module
nvidia-driver-ctr	nvidia, nvidia_modeset, nvidia_uvm
nvidia-peermem-ctr	nvidia_peermem
nvidia-gdrcopy-ctr	gdrdrv

All four dependent modules (nvidia_modeset, nvidia_uvm, nvidia_peermem, gdrdrv) hold references on the nvidia module. However, _unload_driver() only counts three of them in nvidia_deps. When the driver container restarts (e.g. due to upgrade, pod eviction, or node reboot), it calls _unload_driver() while gdrdrv is still loaded by the co-located gdrcopy sidecar. The nvidia module refcount (4) exceeds the expected nvidia_deps count (3), and the function bails out with:

Could not unload NVIDIA driver kernel modules, driver is in use

The driver container then enters CrashLoopBackOff. The only recovery is to delete the entire driver pod (which terminates the gdrcopy sidecar and unloads gdrdrv), but the issue recurs on any subsequent driver container restart.

Observed in production across multiple clusters running H100/H200 GPU nodes with InfiniBand and GDRCopy enabled.

Fix

Add gdrdrv to the set of known nvidia-dependent modules in _unload_driver(), matching the existing pattern used for nvidia_modeset, nvidia_uvm, and nvidia_peermem:

if [ -f /sys/module/gdrdrv/refcnt ]; then
    rmmod_args+=("gdrdrv")
    ((++nvidia_deps))
fi

When gdrdrv is present, it is added to rmmod_args (so it gets unloaded) and nvidia_deps is incremented (so the refcount check passes). When gdrdrv is not present (no GDRCopy), the block is a no-op.

Applied to all 11 nvidia-driver scripts across ubuntu22.04, ubuntu24.04, ubuntu26.04, rhel8, rhel9, and rhel10 (both compiled and precompiled variants).

Reproduction

1. Deploy GPU Operator with driver.rdma.enabled: true (enables peermem + gdrcopy sidecars)
2. Verify gdrdrv is loaded: lsmod | grep gdrdrv → nvidia refcnt=4
3. Delete the driver container (not the pod) or trigger a driver restart
4. Driver container enters Init:CrashLoopBackOff with "Could not unload NVIDIA driver kernel modules"

Testing

• With fix: driver container restart successfully unloads gdrdrv along with the other modules, nvidia refcount check passes, clean reinstall
• Without GDRCopy: gdrdrv sysfs entry doesn't exist, the new block is skipped entirely — no behavior change

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[tariq1890]

The gdrdrv module unload is performed by the nvidia-gdrcopy-ctr sidecar container. As noted by you earlier, a complete driver pod delete unblocks you. Can you go into more details on the exact issue you're facing?

[shaq918]

Thanks for taking a look @tariq1890. The failing scenario is a container-level restart of nvidia-driver-ctr while the pod keeps running — pod deletion isn't involved until someone performs one manually as a recovery step.

Sequence (restartPolicy: Always restarts only the failed container):

1. Driver pod healthy: nvidia-driver-ctr + nvidia-gdrcopy-ctr running; gdrdrv loaded, holding one reference on nvidia.
2. nvidia-driver-ctr alone restarts (liveness probe kill, transient Fabric Manager startup failure, entrypoint crash, …). The gdrcopy sidecar is untouched — kubelet has no reason to restart it — so gdrdrv stays loaded.
3. The restarted driver container runs _unload_driver(): nvidia refcnt exceeds nvidia_deps by exactly gdrdrv's reference → Could not unload NVIDIA driver kernel modules, driver is in use → exit 1 → CrashLoopBackOff.
4. This never self-heals: the sidecar never exits, so gdrdrv is never released. Manual pod deletion is required on every occurrence.

I reproduced this cleanly today (gpu-operator v25.10.1, driver:570.133.20-ubuntu22.04, gdrcopy gdrdrv:v2.5.1, kernel 5.15.0-181). Before the trigger the GPU was idle — no processes held /dev/nvidia*, and module refs were exactly:

/sys/module/nvidia/refcnt:3          # = uvm + modeset + gdrdrv
/sys/module/nvidia_uvm/refcnt:0
/sys/module/nvidia_modeset/refcnt:0
/sys/module/gdrdrv/refcnt:0

Trigger: stopped only the driver container (crictl stop, equivalent to a probe kill or crash). Result:

$ kubectl get pod -n nvidia-gpu-operator
nvidia-gpu-driver-ubuntu22.04-5869f89549-cwv8s   1/2   CrashLoopBackOff   5 (94s ago)   27m
    nvidia-driver-ctr:  restarts=5 lastExit=1
    nvidia-gdrcopy-ctr: restarts=0            # sidecar untouched, still holds gdrdrv

$ kubectl logs -c nvidia-driver-ctr
Unloading NVIDIA driver kernel modules...
nvidia_modeset       1708032  0
nvidia_uvm           1748992  0
nvidia              11522048  3 nvidia_uvm,gdrdrv,nvidia_modeset
Could not unload NVIDIA driver kernel modules, driver is in use

The excess reference is gdrdrv alone (refcnt 3 vs the expected 2). After rmmod gdrdrv from the host — which is what this PR automates — the very next retry unloaded, rebuilt, and reinstalled the driver successfully.

The change follows the existing precedent in the same function: nvidia-peermem is also owned by a sidecar, yet _unload_driver() rmmods it and counts it in nvidia_deps for exactly this reason. gdrdrv is the only sidecar-loaded module missing from that handling. (nvidia_fs from the GDS sidecar looks like it has the same gap — happy to extend the PR if you'd like it in scope.)

One related observation from the repro: after recovery, gdrdrv remained unloaded — the gdrcopy container only has a startupProbe, so unlike peermem (whose probe_nvidia_peermem livenessProbe restarts the sidecar to re-modprobe), nothing reloads gdrdrv until the pod is recreated. So the complete story probably wants a companion liveness probe on nvidia-gdrcopy-ctr in gpu-operator, mirroring the peermem pattern — I'm happy to open that as a follow-up PR if you agree with the direction.