[Feature]: Persist MIG mode and SR-IOV vGPU VFs across reboot on the sandbox/vGPU path (operator-driven GPU reset)

[lexfrei]

Motivation

On the sandbox / vGPU workload path, changing MIG mode and creating SR-IOV vGPU VFs does not survive a node reboot, and the operator does not re-establish this state on boot. After a reboot of an NVSwitch node configured for MIG-backed vGPU:

• MIG mode / MIG instances are not present until re-applied;
• SR-IOV VFs (created via sriov-manage -e) are gone — VFs are runtime state and are never persisted by SR-IOV;
• the vGPU devices created on those VFs are gone.

The operator's MIG manager can enable MIG mode, but committing a MIG-mode change on these GPUs requires a GPU reset. The only reset mechanism the Kubernetes mig-manager has is WITH_REBOOT ("reboot the node if changing the MIG mode fails for any reason") — it never performs a targeted nvidia-smi --gpu-reset. And VF creation (sriov-manage) is not performed by the operator at all — sriov-manage appears nowhere in gpu-operator. So on the vGPU path, after a reboot, MIG + VFs + vGPUs have to be re-established out-of-band (host units that enable MIG, run --gpu-reset, run sriov-manage -e per PF, then release the operand readiness gate).

By contrast, the systemd deployment of mig-parted does handle reboot persistence: nvidia-mig-manager.service persists the selected config across reboot (persist_config_across_reboot) and orders itself via nvidia-gpu-reset.target (After=nvidia-fabricmanager.service, Before=nvidia-gpu-reset.target) so MIG reconfiguration, GPU reset, and driver-service ordering are coordinated at boot. The Kubernetes / operator path has no equivalent.

Proposal / discussion

1. Give the operator a way to commit MIG-mode changes via a targeted GPU reset instead of only a full node reboot (WITH_REBOOT). This likely needs coordination with nvidia-persistenced (see nvidia-persistenced keeps a handle on the GPU and blocks --gpu-reset #118), which holds a handle that blocks --gpu-reset.
2. On the vGPU / sandbox path, have the operator own re-establishing SR-IOV VFs on boot (the equivalent of sriov-manage -e per PF) and re-applying the MIG + vGPU device configuration, so a reboot recovers hands-off without host units. This is the smallest useful seam and could be a first change on its own (VF re-enablement in the vGPU device-manager operand before it applies the vGPU config).
3. Port the systemd deployment's boot-ordering guarantees (config persistence + nvidia-gpu-reset.target ordering) to the operator's operands, or document the boot units as a supported companion for the Kubernetes path.

Related

• nvidia-device-plugin-validator fails after node reboot (with MIG enabled) #403 — device-plugin validator failing after reboot with MIG enabled.
• nvidia-persistenced keeps a handle on the GPU and blocks --gpu-reset #118 — nvidia-persistenced keeps a handle on the GPU and blocks --gpu-reset.

This is the reboot-recovery glue the operator lacks on the vGPU path today.

Environment

NVSwitch HGX 8-GPU node, MIG-backed and whole-card vGPU, host-installed driver, GPU Operator sandbox path (gpu.workload.config=vm-vgpu).

[rajathagasthya]

Thanks for the detailed write-up @lexfrei!

Give the operator a way to commit MIG-mode changes via a targeted GPU reset instead of only a full node reboot

By the way, this is tracked in NVIDIA/mig-parted#357. I believe the right place to do a targeted reset is MIG manager rather than the validator.

Port the systemd deployment's boot-ordering guarantees (config persistence + nvidia-gpu-reset.target ordering) to the operator's operands, or document the boot units as a supported companion for the Kubernetes path.

Could you expand more on this? We already have some ordering for the operands: driver ready -> toolkit ready -> other operands so MIG operations are only performed when that ordering is satisfied.

I'm not too familiar with VFs and how it impacts MIG configuration, but I can discuss internally with the team and get back to you.

[lexfrei]

Thanks! NVIDIA/mig-parted#357 makes sense as the home for a targeted reset primitive — agreed the validator isn't its natural owner. One constraint for the placement discussion: on the sandbox/vm-vgpu path the MIG manager operand isn't scheduled — MIG geometry there is driven by vgpu-device-manager — so if mig-parted grows a targeted per-GPU reset, the consumer on the vGPU path would likely be vgpu-device-manager (or the sandbox validator, which is where #2603 put it for lack of a better hook). Happy to rework #2603 against whatever primitive lands from NVIDIA/mig-parted#357.

On the boot-ordering question: the driver → toolkit → operands readiness chain is necessary but sits above the layer where the vGPU path breaks. After a node reboot, nothing GPU-side survives on our system: MIG mode is uncommitted, the SR-IOV VFs are gone, and the vGPU devices with them. Getting back to a serving state requires a strict order within the GPU state:

1. enable MIG mode on the MIG-backed cards and commit it via GPU reset — the reset requires SR-IOV disabled, so it must precede VF enable;
2. enable the VFs (sriov-manage -e) — and only then
3. create the MIG GPU instances — we found (H200, vgpu-kvm 580.x) that toggling SR-IOV wipes existing GPU instances, so anything carved before VF-enable is destroyed;
4. write the vGPU types to the VFs;
5. start (or rescan) the device plugin — it scans once at startup, so it must come last (or rediscover: feat(device_plugin): rediscover vendor-specific VFIO vGPU profiles at runtime kubevirt-gpu-device-plugin#194).

Steps 1–4 carry hardware-enforced inter-step constraints that operand-level readiness ordering can't express — that's what our host systemd units encode (the equivalent of the systemd deployment's nvidia-gpu-reset.target ordering plus config persistence). "Port to the operator" concretely means some operand on the sandbox path owning this sequence after boot; "document as companion" means blessing boot units like ours as the supported Kubernetes-path answer. Either resolves the gap — glad to help with whichever direction the team lands on.

[lexfrei]

To move the direction question forward, a concrete proposal:

1. The targeted per-GPU reset primitive lands in mig-parted ([Question]: Mig-Manager GPU reset behavior? mig-parted#357), replacing the all-GPU util.ResetAllGPUs with a device-scoped reset — that's its natural home, it owns MIG state transitions today.
2. On the container path, k8s-mig-manager consumes it as-is.
3. On the sandbox/vm-vgpu path — where the MIG manager operand isn't scheduled — the consumer would be vgpu-device-manager: after applying a vGPU config that requires a MIG-mode change, it calls the primitive to commit the mode on exactly the affected GPUs, gated the way feat(validator): commit pending MIG-mode change via targeted GPU reset #2603 gates it today (uncommitted-enable only, no VFs, no compute processes).
4. feat(validator): commit pending MIG-mode change via targeted GPU reset #2603 then shrinks to wiring that call into the sandbox path, and the validator stops owning resets.

If that direction sounds right, I'd rework #2603 accordingly once the primitive exists — happy to contribute the mig-parted primitive too.