feature(cluster): Add VULTR vke as an inference cluster provider

[haarchri]

Description of your changes

Modelplane can provision inference clusters on GKE, EKS, AKS, and Nebius; Vultr was listed as Planned on the providers page. This adds Vultr VKE as a fifth provisioned cluster source, built on the xpkg.upbound.io/upbound/provider-vultr provider.

An InferenceCluster with source: Vultr composes a new VultrCluster XR, served by the new compose-vultr-cluster function. The VKE cluster carries a fixed inline system pool for control-plane components; each user GPU pool becomes a separate VkeNodePool managed resource gated on cluster readiness, so pools can be added, resized, or removed without touching the cluster. InferenceClass gains a vultr provisioning block. Vultr sizes nodes by plan (e.g. vcg-l40s-16c-180g-48vram for 1x L40S), so the block is just plan plus accelerator.

Two things VKE ships built in shape the composition. Vultr pre-installs the NVIDIA GPU Operator on GPU clusters, so rather than composing a GPU stack, VultrCluster gates its readiness on an Observe-only Object with a CEL query over the nvidia-operator-validator DaemonSet, the cluster only reports Ready once the GPU stack is validated and nvidia.com/gpu is advertised, and the serving stack never starts before that. The Vultr CSI driver likewise installs the vultr-vfs-storage RWX StorageClass (Vultr File System) on every cluster, and node autoscaling is served by VKE itself, so no CSI driver, StorageClass, or in-cluster autoscaler is composed.

VKE ships the vultr-vfs-storage ReadWriteMany StorageClass backed by Vultr File System, but it is not usable on GPU nodes, so ModelCache RWX storage is served by Longhorn instead, a Helm release installs Longhorn (tolerating the GPU taint so its node components run everywhere), an NFS-client installer DaemonSet satisfies Longhorn's RWX prerequisite on VKE's node image, and the modelplane-rwx-longhorn StorageClass pins ModelCache PVCs to driver.longhorn.io.

Known limitation:

• UED-84YHJ: fabric networking is not available for GPU nodes either, so there is no high-bandwidth interconnect for multi-node serving. Serving is therefore limited to single-node GPU instances#

Note:

• YTM-97SIL: VKE does not support VPC-only clusters (the vpc_only flag fails cluster creation with Error 500), so nodes keep public network interfaces; a VPC can be attached but not made the only network; thats how Vultr API works today

Fixes #305

I have:

• Read and followed Modelplane's contribution process.
• Run nix flake check (or ./nix.sh flake check) and made sure it passes.
• Added or updated tests covering any composition function changes.
• Signed off every commit with git commit -s.

Tested

Validated end to end: a single-node L40S GPU pool provisions, passes the GPU readiness gate, and serves a model.

• Create ProviderConfig + Secret

apiVersion: v1
kind: Secret
metadata:
  name: vultr-credentials
  namespace: crossplane-system
type: Opaque
stringData:
  api-key: ${VULTR_API_KEX}
---
apiVersion: vultr.crossplane.io/v1alpha1
kind: ClusterProviderConfig
metadata:
  name: default
spec:
  credentials:
    source: Secret
    secretRef:
      namespace: crossplane-system
      name: vultr-credentials
      key: api-key

• Create Modelplane Manifests:

---
apiVersion: modelplane.ai/v1alpha1
kind: InferenceGateway
metadata:
  name: default
  namespace: crossplane-system
spec:
  backend: Traefik
  traefik:
    version: "40.2.0"
    loadBalancer: MetalLB
    metallb:
      addressPool: "172.18.255.200-172.18.255.250"
---
apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: a16-1x
  namespace: crossplane-system
spec:
  description: "Vultr vcg-a16-6c-64g-16vram, NVIDIA A16 16GB"
  provisioning:
    provider: Vultr
    vultr:
      plan: vcg-a16-6c-64g-16vram
      accelerator:
        type: nvidia-a16
        count: 1
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 1
    attributes:
      architecture: { string: Ampere }
      cudaComputeCapability: { version: "8.6.0" }
    capacity:
      memory: { value: "16384Mi" }
---
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: vultr-demo
  namespace: crossplane-system
  labels:
    modelplane.ai/region: ewr
spec:
  cluster:
    source: Vultr
    vultr:
      region: ewr
  nodePools:
  - name: gpu-a16
    className: a16-1x
    nodeCount: 1
    minNodeCount: 1
    maxNodeCount: 1
---
apiVersion: v1
kind: Namespace
metadata:
  name: ml-team
---
apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: qwen-7b
  namespace: ml-team
spec:
  replicas: 1
  template:
    spec:
      engines:
      - name: qwen-7b
        members:
        - role: Standalone
          nodeSelector:
            devices:
            - name: gpu
              count: 1
              selectors:
              - cel: |
                  device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("14Gi")) >= 0
          template:
            spec:
              containers:
              - name: engine
                image: vllm/vllm-openai:v0.9.2
                args:
                - --model=Qwen/Qwen2.5-7B-Instruct-AWQ
                - --served-model-name=qwen-7b
                - --max-model-len=8192
                - --gpu-memory-utilization=0.85
                - --enforce-eager
---
apiVersion: modelplane.ai/v1alpha1
kind: ModelService
metadata:
  name: qwen-7b
  namespace: ml-team
spec:
  endpoints:
  - selector:
      matchLabels:
        modelplane.ai/deployment: qwen-7b

• Tested Model + ModelEndpoint:

{"id":"chatcmpl-5f3c5820-b281-4872-8ed0-9137b2890359","object":"chat.completion","created":1785133519,"model":"qwen-7b","choices":[{"index":0,"message":{"role":"assistant","reasoning_content":null,"content":"Hello from Vultr! It's nice to meet you. Vultr is a cloud computing platform that offers flexible and affordable virtual server instances. How can I assist you today with regards to Vultr or cloud computing in general?","tool_calls":[]},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":35,"total_tokens":81,"completion_tokens":46,"prompt_tokens_details":null},"prompt_logprobs":null,"kv_transfer_params":null}

[haarchri]

after chatting with @mayankdebnath we added longhorn for ModelCache StorageClass - that we can use storage with GPU Nodes ... its working as validated:

8s           Normal  SuccessfulAttachVolume  Pod/qwen-7b-bf743-qwen-7b-a7b89-69b8dc8d49-plw89       AttachVolume.Attach succeeded for volume "pvc-66369d5e-899e-47d7-9382-8c72d5fe0fd1"
7s           Normal  Pulling          Pod/qwen-7b-bf743-qwen-7b-a7b89-69b8dc8d49-plw89       Pulling image "vllm/vllm-openai:v0.9.2"

[negz]

Should this provider be in crossplane-contrib if it's not official? 🤔

[haarchri]

we have the provider-vultr in upbound org for now, we need to figure out which provider we will use on the long-run https://github.com/vultr/crossplane-provider-vultr (upjet based but no official release since years and behind the TF version), https://github.com/crossplane-contrib/provider-vultr (native, archived since 2025) and our published provider https://github.com/upbound/provider-vultr (native)

[negz]

I'm not familiar with Longhorn. I assume it could work for other providers that don't have native support for RWX PVs? Does it warrant a small design before we choose and commit to it? Are there alternatives we should consider?

I'm asking because I want to make sure we extend our tech stack intentionally. If Longhorn is the right choice I'm all for it, but I want to make sure we thought it through.

[negz]

I did a little research. Any idea what its performance is like?

Seems like there could be some quirks, namely:

• I think it allocates up to 70% of node disk for storage, so the model weights need to fit in that.
• The NFS share-manager funnels all traffic through a single node - so N engines will all load from one node, capped at that node's network bandwidth.

(I'm trusting an agent's analysis here, so I could be wrong.)

[haarchri]

The 70% is storage-reserved-percentage-for-default-disk (default 30% reserved). It is a setting, not a hard limit - and we using here bare metal nodes with lot of space

The share-manager funnel is real. Longhorn attaches the volume RWO to one node and re-exports it over NFS. All engines read through that one pod, capped at its NIC. It is per volume, so one PVC per model spreads it a bit...

The thing is, this is not really a Longhorn problem. AWS EFS for example caps at 500 MiB/s to 1.5 GiB/s per client, and in Bursting mode a 200 GB filesystem drops to 10 MiB/s once credits run out. Vultr does have native RWX (VFS), but on bare metal it goes through a single Storage Gateway, same shape as share-manager. Every option we have funnels somewhere....

So what we actually need is a bar: N engines, model size S, ready within T. Whatever backend cannot hit it does not ship, whether it is Longhorn, EFS or VFS. If none of them clear it, that tells us to stop using it and start thinking of something different instead?

I would treat Longhorn as an implementation detail for now, not a commitment. It sits behind the ModelCache backend and I expect we replace it. If VFS covers our gpu node pools we can drop it ...

[negz]

The thing is, this is not really a Longhorn problem.

Isn't being capped to a single node's NIC speed a problem?

We know with EFS that we hit approx 1.4GB/s per engine load, with two engines loading simultaneously:

modelplane/docs/manifests/examples/kimi-k2/model-deployment.yaml

Lines 14 to 15 in 74b8930

	# --load-format=runai_streamer cold-reads ~509 GiB per engine off the shared
	# RWX cache in ~6 minutes (vs ~45 with the default loader).

I believe there's a 20GB/s limit at the sender end, so we could have ~14 engines loading simultaneously before we started to see that slow down.

We also know (per #204) that vLLM can pull directly from HuggingFace at around 500MB/s (per engine).

So what we actually need is a bar: N engines, model size S, ready within T.

That's a good idea, and the kind of thing stepping back and thinking about design would help us align on.

For now, what I want to avoid is adding ModelCache support that actually makes start time slower in most cases. I can't find any published NIC speeds for Vultr, but assuming 10G NICs that's about about a 1GB/s ceiling for all engines to read from cache, right? So with 2 engines it's about as fast as loading directly from HF, and over 2 engines it's slower? Happy to be wrong here - I don't have access to test so I'm estimating.

I don't think ModelCache support necessarily needs to block Vultr support. If Longhorn is fast enough to be useful, great. If not, I'd rather ship Vultr without ModelCache support until we have a better cache solution.

[haarchri]

Agreed, and I am fine dropping Longhorn here. It was meant as an implementation detail...

I would rather split this into two things.

1. Vultr ships without ModelCache. Better to ship a provider with a documented gap. This connects to Keep cluster provisioning implementations consistent as we add providers #331, if we write a contract for what every provider implementation must deliver, I think "cache StorageClass" should be optional and declared, not mandatory. Some clouds will not have a good RWX story for a while, and blocking the whole provider on it means we ship nothing. What matters is that the gap is visible in the docs and in the API, so users know what they get.

2. A separate design for shared model storage. Not "which RWX backend", the wider question.

• The bar, measured. N engines, model size S, ready within T. With real numbers from real clusters, not estimates. Right now we are all guessing.
• Is a shared filesystem the right shape at all? Every RWX option funnels through one point somewhere. So we would only be choosing between funnels.
• HF in the scale-up path. If every scale-out pulls from HF, then HF rate limits or an HF outage means no scale-out. That should be a deliberate decision...
• Traffic cost. Without a cache we pull S per engine on every cold start and every scale-out. That is billed, per GB read on Storage, egress elsewhere. A cache can be slower on wall clock and still be right if it turns N pulls into 1