feat(cocoonset): explicit not-ready/unreachable tolerations on managed pods#19
Closed
tonicmuroq wants to merge 1 commit into
Closed
feat(cocoonset): explicit not-ready/unreachable tolerations on managed pods#19tonicmuroq wants to merge 1 commit into
tonicmuroq wants to merge 1 commit into
Conversation
…d pods Managed pods only tolerated the cocoon taint, so DefaultTolerationSeconds admission injected 300s not-ready/unreachable NoExecute tolerations. Any control-plane window longer than ~6min (node NotReady grace + 300s) let taint eviction delete the pods for real, and vk-cocoon then faithfully removed the backing VMs. For stateful pinned VMs eviction is never a valid recovery: a recreated pod clones a fresh VM from the base image under the old VMName (snapshots are only consulted by the explicit restore flow), so the automatic path can only ever produce garbage data under a stale identity. - pods.go: newManagedPod stamps explicit Exists/NoExecute tolerations for node.kubernetes.io/not-ready and node.kubernetes.io/unreachable, so the admission default no longer applies - EVICTION_TOLERATION_SECONDS env: 0 or unset (default) tolerates forever; >0 bounds the toleration for pools where eviction is acceptable; invalid values fail fast at startup - deployment.yaml: document the knob with its default - pods_test.go: pin the forever default and the bounded variant Existing pods keep their current tolerations (drift detection ignores the field, so no recreate churn); tolerations are append-only mutable and can be patched onto the running fleet separately.
Contributor
Author
|
close as 748e3f3 did a stricter implementation |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Managed pods only tolerate the cocoon taint, so the DefaultTolerationSeconds admission plugin injects the default 300s
node.kubernetes.io/not-ready/node.kubernetes.io/unreachableNoExecute tolerations. Any window where a cocoon node stays NotReady longer than ~6 minutes (node-monitor grace + 300s) — a control-plane partition, a long vk-cocoon maintenance stop, an upgrade gone slow — ends with taint eviction deleting the pods for real, and vk-cocoon then faithfully removing the backing VMs once it syncs.For stateful, host-pinned VMs eviction is never a valid recovery mechanism:
The correct semantic is: pods on a dead/unreachable node wait in place; recovery is an explicit per-VM decision (restore from snapshot / declare loss).
What
newManagedPodstamps explicitExists/NoExecutetolerations fornode.kubernetes.io/not-readyandnode.kubernetes.io/unreachable(admission only injects its 300s defaults when the pod has none).EVICTION_TOLERATION_SECONDSenv on the operator: 0 or unset (default) = tolerate forever;>0bounds the toleration for pools where eviction is acceptable; invalid values fail fast at startup.config/manager/deployment.yamldocuments the knob.Notes
podSpecMatchesAgentignores tolerations, so no recreate churn on rollout. Tolerations are append-only mutable on running pods; the live fleet can be patched separately.go build ./... && go vet ./... && go test ./...green.