[Bug]: When autoscale GPU to zero, MIG configuration failed if mig.config is set before the node starts

[mickours]

Describe the bug

I have a cluster with configured H100 GPU node pools scaled to zero, one per MIG configuration, with predefined labels for the MIG. For example, nvidia.com/mig.config: "all-1g.10gb" is set on the node by the cloud provider (here Scaleway). So when it starts, the label is already present.

When a pod triggers the autoscaling, the node starts, but I have two issues:

1. The pod starts before the MIG was properly set, and thus, it uses the full GPU
2. When the nvidia-device-plugin-daemonset starts, it fails with this error:

E0723 19:02:59.909633       1 main.go:182] error starting plugins: error getting plugins: unable to create plugins: failed to construct resource managers: error building device map: error building device map from config.resources: invalid MIG configuration: at least one device with migEnabled=true was not configured correctly: error visiting device: device 0 has no MIG devices configured

When I force the mig.config label to another profile and then come back to the one I want it does work propely.

To Reproduce

Create a GPU node pool scaled to zero with preset labels nvidia.com/mig.config: "all-1g.10gb" and create a pod that triggers the autoscaler

Expected behavior

The GPU operator should detect earlier that the mig.config label is set and change the MIG profile accordingly (without errors) and disallow other pods to be scheduled (for example, using a taint).

Environment (please provide the following information):

• GPU Operator Version: v25.10.1
• OS: ubuntu 24.04.4 LTS - 2026-07-10
• Kernel Version: 6.8.0-124-generic (amd64)
• Container Runtime Version: containerd 2.3.1
• Kubernetes Distro and Version: Scaleway Kapsule 1.36.1

Information to attach (optional if deemed irrelevant)

Full logs of the device plugin when it fails

``` IS_HOST_DRIVER=false NVIDIA_DRIVER_ROOT=/run/nvidia/driver DRIVER_ROOT_CTR_PATH=/driver-root NVIDIA_DEV_ROOT=/run/nvidia/driver DEV_ROOT_CTR_PATH=/driver-root Starting nvidia-device-plugin I0723 19:02:59.852813 1 main.go:244] "Starting NVIDIA Device Plugin" version=< dfcf7283 commit: dfcf72838b8839c7a0fd9bc9d21fe8d46bed2111 > I0723 19:02:59.852853 1 main.go:247] Starting FS watcher for /var/lib/kubelet/device-plugins I0723 19:02:59.852889 1 main.go:254] Starting OS watcher. I0723 19:02:59.853137 1 main.go:269] Starting Plugins. I0723 19:02:59.853151 1 main.go:326] Loading configuration. I0723 19:02:59.853788 1 main.go:352] Updating config with default resource matching patterns. I0723 19:02:59.853970 1 main.go:363] Running with config: { "version": "v1", "flags": { "migStrategy": "single", "failOnInitError": true, "mpsRoot": "/run/nvidia/mps", "nvidiaDriverRoot": "/run/nvidia/driver", "nvidiaDevRoot": "/run/nvidia/driver", "gdrcopyEnabled": false, "gdsEnabled": false, "mofedEnabled": false, "useNodeFeatureAPI": null, "deviceDiscoveryStrategy": "auto", "plugin": { "passDeviceSpecs": true, "deviceListStrategy": [ "cdi-annotations", "cdi-cri" ], "deviceIDStrategy": "uuid", "cdiAnnotationPrefix": "cdi.k8s.io/", "nvidiaCTKPath": "/usr/local/nvidia/toolkit/nvidia-cdi-hook", "containerDriverRoot": "/driver-root" } }, "resources": { "gpus": [ { "pattern": "*", "name": "nvidia.com/gpu" } ], "mig": [ { "pattern": "*", "name": "nvidia.com/gpu" } ] }, "sharing": { "timeSlicing": {} }, "imex": {} } I0723 19:02:59.853980 1 main.go:366] Retrieving plugins. time="2026-07-23T19:02:59Z" level=info msg="Using /driver-root/usr/lib/x86_64-linux-gnu/libnvidia-sandboxutils.so.580.105.08" time="2026-07-23T19:02:59Z" level=info msg="Auto-detected mode as 'nvml'" E0723 19:02:59.909633 1 main.go:182] error starting plugins: error getting plugins: unable to create plugins: failed to construct resource managers: error building device map: error building device map from config.resources: invalid MIG configuration: at least one device with migEnabled=true was not configured correctly: error visiting device: device 0 has no MIG devices configured ```

[rahulait]

Thanks for the report @mickours . We'll take a look.

[rajathagasthya]

@mickours This is a known gap with Autoscalers (#708) we're actively addressing.

Between node ready and MIG configuration completing, nothing gates scheduling, so the workload pod can land on the un-partitioned GPU and block MIG setup. I think what's happening here is MIG fails to configure profiles (enables MIG, but instance creation fails) because the workload is holding the full GPU. MIG Manager then restores the device-plugin after the failure, which then finds MIG is enabled but has zero devices and crash loops. We can probably confirm this with MIG Manager logs. The mig.config label change works for you because it re-triggers mig-manager's reconfiguration after the offending pod is gone.

Presetting nvidia.com/mig.config in the pool template is correct, but the missing piece is exactly the taint mechanism. We're publishing a reference architecture on how to use GPU Operator with Autoscalers (startup taint + Node Problem Detector + Node Readiness Controller) in #2573. For MIG pools, the readiness probe should gate on MIG state, which that guide covers.