Skip to content

[Bug]: GCP fractional G4 instances are exposed as full 96GB GPUs #4039

Description

@peterschmidt85

Steps to reproduce

  1. Create a fleet pinned to g4-standard-6:
type: fleet
name: gcp-g4-fractional

nodes: 1
backends: [gcp]
regions: [us-central1]
instance_types: [g4-standard-6]
spot_policy: spot

resources:
  gpu: RTXPRO6000:1
  1. Run a minimal CUDA task on that fleet:
type: task
name: gcp-g4-fractional-cuda

image: nvidia/cuda:13.0.3-base-ubuntu24.04
commands:
  - ls -l /dev/nvidia*
  - nvidia-smi

fleets: [gcp-g4-fractional]
creation_policy: reuse

resources:
  gpu: RTXPRO6000:1

Actual behaviour

The plan reports g4-standard-6 as RTXPRO6000:96GB:1. Google documents this machine type as a fractional 1/8 GPU with 12GB of GPU memory.

The instance boots dstack-cuda-0-14, which uses the open NVIDIA kernel modules. The host kernel rejects the G4 vGPU:

NVRM: The NVIDIA vGPU 0000:00:03.0 (PCI ID: 10de:2bb5)
NVRM: installed in this system is not supported by open nvidia.ko.
nvidia 0000:00:03.0: probe with driver nvidia failed with error -1
NVRM: None of the NVIDIA devices were initialized.

The shim logs host nvidia-smi exit code 9 and detects no GPUs. The whole-instance job still starts without an NVIDIA Docker device request, so the task has no /dev/nvidia0 and nvidia-smi is not injected.

This also lets g4-standard-6 incorrectly satisfy requirements above its real 12GB GPU memory.

Expected behaviour

Until fractional G4 vGPUs are explicitly supported, dstack must not return or provision:

  • g4-standard-6 (1/8 GPU, 12GB)
  • g4-standard-12 (1/4 GPU, 24GB)
  • g4-standard-24 (1/2 GPU, 48GB)

Full-GPU G4 machine types must remain available. A control smoke on g4-standard-48 passed with the same dstack VM image and CUDA container: host driver 580.167.08, CUDA 13.0, and 97,887 MiB visible in the container.

dstack version

0.20.27 with gpuhunt 0.1.26

Additional information

The original G4 support PR documented g4-standard-48; the GA PR listed only g4-standard-48, g4-standard-96, g4-standard-192, and g4-standard-384. Fractional G4 support was not intended.

Suggested immediate fix:

  1. Filter g4-standard-6, g4-standard-12, and g4-standard-24 from GCP offers in dstack, with a focused test that keeps g4-standard-48 supported.
  2. Fix gpuhunt so unsupported fractional shapes are not published as full RTXPRO6000:96GB GPUs.
  3. Track actual fractional-vGPU support separately. It requires effective memory reporting, GCP's vGPU/GRID driver, and VM identity authentication; globally classifying all RTXPRO6000 GPUs as proprietary-driver devices would be incorrect.

Google references:

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions