Harden reboot-required and reboot pods - #44
Open
moberghammer wants to merge 1 commit into
Open
Conversation
The utility pods were bare best-effort pods pinned via spec.nodeName, which caused three failure modes seen during full-fleet reboot rounds: - spec.nodeName bypasses the scheduler, so on a node at max-pods capacity the kubelet rejects the pod outright (OutOfpods, phase Failed) and preemption never gets a chance to run. During a serial fleet round the not-yet-rebooted nodes absorb the drained workloads and hit the pod cap exactly when their probe is due, stalling the round. The pods are now placed through the scheduler with a required node affinity on metadata.name, and an optional reboot.podPriorityClassName config lets them preempt lower-priority pods when the node is full. - The pods declared no controller, so a leftover probe pod blocked the node's own drain (the drain helper refuses pods without a controller). They now carry a controller ownerReference to their target Node, like kubelet mirror pods, which also garbage-collects them if the node object is deleted. - No resource requests/limits, which fails common admission policies (e.g. Kyverno require-requests-limits) and made the pods first in line for kubelet rejection. Both pods now request 10m/16Mi with 100m/32Mi limits. Also replaces the deprecated container.apparmor.security.beta annotation on the reboot pod with the securityContext.appArmorProfile field (GA in Kubernetes 1.30).
Owner
|
I have added priorityClassName, resources and apparmor in #45 @moberghammer can you create an issue for the drain/garbage collect part, pretty hard to read from the description what the issue you are trying to resolve. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
While running full-fleet reboot rounds (~34 nodes across two clusters) with nodedrain 0.0.10 we hit three failure modes that all trace back to the utility pods (
reboot-required-*,reboot-*) being bare best-effort pods pinned viaspec.nodeName:OutOfpodsprobe failures on full nodes.spec.nodeNamebypasses the scheduler, so kubelet admission rejects the pod outright when the node is at max-pods capacity — preemption never runs. This bites exactly during a serial fleet round: not-yet-rebooted nodes absorb the drained workloads, hit the pod cap, and then their own reboot-required probe fails (phase: Failed,reason: OutOfpods), stalling the round until an operator frees pod slots.cannot delete Pods that declare no controller).require-requests-limits), and best-effort pods are first in line for kubelet rejection.Changes:
metadata.name(the existing tolerations already cover cordoned/control-plane nodes). A new optional configreboot.podPriorityClassName(default"", unchanged behavior) lets operators point at a PriorityClass so the pods can preempt lower-priority pods on full nodes.container.apparmor.security.beta.kubernetes.io/shellannotation on the reboot pod is replaced withsecurityContext.appArmorProfile(field is GA since Kubernetes 1.30).Trade-offs worth calling out: pod placement now depends on a working scheduler, and clusters running the
OwnerReferencesPermissionEnforcementadmission plugin may need the controller to hold delete permission on nodes for the ownerRef.make testpasses (envtest, k8s 1.34).