Skip to content

[Feature]: Support affinity / nodeSelector for operand DaemonSets in ClusterPolicy spec.daemonsets #2626

Description

@yalattas

Feature description

Add affinity and nodeSelector to the common DaemonSet config in ClusterPolicy
(spec.daemonsets), surfaced in the Helm chart as daemonsets.affinity /
daemonsets.nodeSelector (default empty). When set, apply them to all operand
DaemonSets (driver, container-toolkit, device-plugin, dcgm-exporter, gfd,
mig-manager, …), on top of the operator's existing NFD-based node selection.

Motivation

spec.daemonsets already exposes tolerations but has no affinity/nodeSelector.
That's an asymmetry: tolerations only permit scheduling onto tainted nodes — they
can't restrict where operands land. On a cloud cluster (EKS/GKE/AKS) with several
autoscaled node pools where only one has GPUs, I want to declaratively pin operands to
that pool via a label it already carries (e.g. karpenter.sh/nodepool=gpu).

The documented alternatives don't cover this well:

  • nvidia.com/gpu.deploy.<operand>=disabled labels / node taints (how Deploy nvidia-device-plugin-daemonset to only certain nodes #547 was
    closed) are subtractive: you must label or taint every node you want to exclude.
    In autoscaled clusters (Karpenter, managed nodegroups) where nodes churn constantly,
    that doesn't scale and is race-prone.
  • NFD gating keeps operands off non-GPU nodes, but can't target a specific GPU
    pool and doesn't apply when NFD is disabled.

A first-class affinity/nodeSelector is declarative, applies automatically to new
nodes, and is consistent with the operator's own NvidiaDriver CRD, which already
exposes spec.affinity and spec.nodeSelector.

Prior art

#547 requested essentially this and was closed pointing at the subtractive workarounds
above; a community MR to add nodeSelector to the common daemonset config was referenced
there but never landed.

Proposed API

# ClusterPolicy
spec:
  daemonsets:
    nodeSelector:            # proposed
      karpenter.sh/nodepool: gpu
    affinity: {}             # proposed
# Helm values.yaml
daemonsets:
  affinity: {}       # default: unchanged behavior
  nodeSelector: {}   # default: unchanged behavior

User-supplied values should be merged (AND) with the operator's NFD-based selection so
GPU detection still applies.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions