Skip to content

Harden reboot-required and reboot pods - #44

Open
moberghammer wants to merge 1 commit into
SlyngDK:mainfrom
cego:harden-utility-pods
Open

Harden reboot-required and reboot pods#44
moberghammer wants to merge 1 commit into
SlyngDK:mainfrom
cego:harden-utility-pods

Conversation

@moberghammer

Copy link
Copy Markdown

While running full-fleet reboot rounds (~34 nodes across two clusters) with nodedrain 0.0.10 we hit three failure modes that all trace back to the utility pods (reboot-required-*, reboot-*) being bare best-effort pods pinned via spec.nodeName:

  1. OutOfpods probe failures on full nodes. spec.nodeName bypasses the scheduler, so kubelet admission rejects the pod outright when the node is at max-pods capacity — preemption never runs. This bites exactly during a serial fleet round: not-yet-rebooted nodes absorb the drained workloads, hit the pod cap, and then their own reboot-required probe fails (phase: Failed, reason: OutOfpods), stalling the round until an operator frees pod slots.
  2. Leftover probe pods block the node's own drain. The pods declare no controller, and the drain helper refuses to evict pods without a controller ownerRef (cannot delete Pods that declare no controller).
  3. Admission-policy warnings. No resource requests/limits fails common policies (e.g. Kyverno require-requests-limits), and best-effort pods are first in line for kubelet rejection.

Changes:

  • Pods are now placed through the scheduler using a required node affinity on metadata.name (the existing tolerations already cover cordoned/control-plane nodes). A new optional config reboot.podPriorityClassName (default "", unchanged behavior) lets operators point at a PriorityClass so the pods can preempt lower-priority pods on full nodes.
  • Both pods get a controller ownerReference to their target Node (same pattern as kubelet mirror pods) — satisfies the drain helper and garbage-collects pods for deleted nodes.
  • Both pods get requests 10m/16Mi, limits 100m/32Mi.
  • The deprecated container.apparmor.security.beta.kubernetes.io/shell annotation on the reboot pod is replaced with securityContext.appArmorProfile (field is GA since Kubernetes 1.30).

Trade-offs worth calling out: pod placement now depends on a working scheduler, and clusters running the OwnerReferencesPermissionEnforcement admission plugin may need the controller to hold delete permission on nodes for the ownerRef.

make test passes (envtest, k8s 1.34).

The utility pods were bare best-effort pods pinned via spec.nodeName,
which caused three failure modes seen during full-fleet reboot rounds:

- spec.nodeName bypasses the scheduler, so on a node at max-pods
  capacity the kubelet rejects the pod outright (OutOfpods, phase
  Failed) and preemption never gets a chance to run. During a serial
  fleet round the not-yet-rebooted nodes absorb the drained workloads
  and hit the pod cap exactly when their probe is due, stalling the
  round. The pods are now placed through the scheduler with a
  required node affinity on metadata.name, and an optional
  reboot.podPriorityClassName config lets them preempt lower-priority
  pods when the node is full.

- The pods declared no controller, so a leftover probe pod blocked
  the node's own drain (the drain helper refuses pods without a
  controller). They now carry a controller ownerReference to their
  target Node, like kubelet mirror pods, which also garbage-collects
  them if the node object is deleted.

- No resource requests/limits, which fails common admission policies
  (e.g. Kyverno require-requests-limits) and made the pods first in
  line for kubelet rejection. Both pods now request 10m/16Mi with
  100m/32Mi limits.

Also replaces the deprecated container.apparmor.security.beta
annotation on the reboot pod with the securityContext.appArmorProfile
field (GA in Kubernetes 1.30).
@SlyngDK

SlyngDK commented Aug 9, 2026

Copy link
Copy Markdown
Owner

I have added priorityClassName, resources and apparmor in #45

@moberghammer can you create an issue for the drain/garbage collect part, pretty hard to read from the description what the issue you are trying to resolve.

@SlyngDK SlyngDK self-assigned this Aug 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants