feat(gcp): slice 4 — node auto-provisioning and the three ComputeClasses - #1822
Merged
Conversation
Narrow on purpose. The design's slice 4 exists to settle ONE question -- criterion 12: does a freshly auto-created node carry `node.cilium.io/agent-not-ready` at registration, and does Cilium clear it? The `io` and GPU classes follow once that is answered; writing three now would mean debugging three variants of the same unknown. TWO HALVES, BOTH REQUIRED opentofu: `cluster_autoscaling` on the GKE module. Without it a ComputeClass with nodePoolAutoCreation still schedules, but only onto pools that already exist -- nothing is ever provisioned and the premise goes untested. The ceiling (criterion 16) is set low deliberately: a too-low limit is an Unschedulable pod and a one-line change, a too-high one is a bill. manifest: one ComputeClass, spot-only across e2 then n2, with `whenUnsatisfiable: DoNotScaleUp` -- criterion 14 requires zero on-demand fallback, so exhausted spot leaves pods Pending rather than quietly becoming expensive. THE PART THAT IS ACTUALLY THE SLICE `spec.nodePoolConfig.taints` carries node.cilium.io/agent-not-ready. The static pool sets that taint through `node_pools_taints` in OpenTofu; an auto-created pool has no OpenTofu to set it, because NAP creates the pool. Anything the pool must carry has to be declared in the ComputeClass or it does not exist. Without it a new node registers Ready before Cilium owns its networking and accepts pods it cannot network -- surfacing as FailedCreatePodSandBox referencing a missing CNI (criterion 13), which points at the CNI rather than at autoscaling. I nearly shipped the class without it; the schema check is what surfaced `nodePoolConfig.taints`. SCHEMA VALIDATION NEEDED A FOURTH CATALOG SOURCE `flux schema validate` runs with skipMissingSchemas: false by design -- an unknown Kind FAILS rather than passing unvalidated -- and ComputeClass is in no public catalog. Unlike the Envoy AI Gateway and Karpenter CRDs, which gen-catalog.sh renders from pinned Helm charts, GKE installs this one itself and publishes no chart, so it is vendored under scripts/flux-schema/vendored-crds/ with its GKE component version recorded in the annotations. It is NOT under crds/ -- Flux applies that tree, and this CRD is reconciled by GKE's addon manager. Applying it would fight GKE. COST (test clusters, spot and cheapest by default) `cluster_autoscaling` left disk on module defaults of 100 GB pd-standard -- TWICE the static pool's 50 GB, purely because the field was unset. Now 50 GB to match, and pd-standard as the cheapest type, with the pd-balanced asymmetry against the static pool noted in place so a later performance surprise has a visible cause. Verified: ./scripts/validate-manifests.sh -> Valid: 1189, Invalid: 0, Skipped: 0, all gates passed -- the ComputeClass is genuinely validated, not skipped; `kubectl apply --dry-run=server` accepts it against the live GKE API; tofu validate and tofu fmt clean; shellcheck -x -S warning clean on the generator. Criterion 12 itself is NOT yet verified -- that needs the cluster to track this branch and actually scale up. Next step, not a claim.
Slice 4 was built to answer one question and answered a second one nobody had
asked. Both are now written down where they will be hit.
MEASURED, NOT ASSUMED (2026-08-24, on gcp-mycluster-0)
Criterion 12 PASSES: an auto-created node carried node.cilium.io/agent-not-ready
at registration, from spec.nodePoolConfig.taints, and Cilium cleared it. After
scale-up the nodes retained only GKE's own cloud.google.com/compute-class taint.
The unasked question: WITHOUT a toleration, nothing scales up at all. Three
Pending pods, ten minutes, and no TriggeredScaleUp event of any kind -- only
FailedScheduling. The autoscaler simulates scheduling against a node that will
carry the class's taint, judges the pod unplaceable, and never provisions one.
Adding ONLY the toleration produced four nodes and all pods Running.
That is a real divergence from AWS, where the same taint is invisible to
workloads: static pools carry it, and Karpenter provisions against NodePool
requirements rather than simulating a tainted node.
DECISION: keep the taint, require the toleration.
It still gates every pod that does not opt in, and keeps auto-created nodes
behaving like static ones. The accepted cost is that a tolerating pod can land
before the Cilium agent is up and log
plugin type="cilium-cni" failed (add): unable to create endpoint ... EOF
which is the CNI PRESENT with its agent still starting, not a missing CNI. It
self-heals: 0 restarts, pods reached Running unaided. kube-system/metrics-server
hits the same transient on any fresh node, so it belongs to the platform rather
than to this decision. Criterion 13 therefore holds as worded -- no pod recorded
a MISSING CNI -- while its spirit is only partly met, and saying so is more
useful than claiming a clean pass.
WHERE IT IS RECORDED
- The ComputeClass itself, in a box at the top of nodePoolConfig, with the
toleration written out ready to copy. A consumer hits this file before they
hit the ADR.
- ADR-0006, as a Negative consequence. Its Positive list claimed Cilium's taint
is "handled declaratively at the pool level", which is true but incomplete in
a way that matters -- declaring the taint works; making it transparent to
workloads does not. That bullet is now amended to point at the negative rather
than left to mislead.
Also observed, unprompted: criterion 14 (e2-highcpu-4, spot=true, zero
on-demand -- NAP chose highcpu over standard, cheaper per vCPU) and criterion 17
(all four auto-created pools removed on scale-down; back to 2 nodes with no
intervention).
Verified: ./scripts/validate-manifests.sh -> Valid: 1189, Invalid: 0,
Skipped: 0, all gates passed; ./scripts/validate-links.sh -> all relative links
resolve.
Completes slice 4's three classes. general-purpose was proven on its own first
because it carried the unknown the slice existed to settle; these two follow the
pattern it established, and each inherits the mandatory Cilium toleration.
MIRRORING AWS WHERE IT IS A CONVENTION, DIVERGING WHERE IT IS A FACT
Taint keys are IDENTICAL to the AWS NodePools -- `ogenki/io` and
`nvidia.com/gpu`. Those are platform conventions rather than cloud details, so a
workload's tolerations are the one part of its manifest that does not change
between clouds, even though everything underneath differs.
gpu-l4 needs NO runtimeClassName, and that is a real divergence rather than an
omission. On AWS the Bottlerocket NVIDIA variant requires
`runtimeClassName: nvidia` plus a matching RuntimeClass
(infrastructure/base/runtimeclass-nvidia/). GKE installs the driver via its own
DaemonSet and wires the default runtime, so a pod just requests nvidia.com/gpu --
which is design criterion 15 exactly: "a GPU pod with NO runtimeClassName sees
the device via nvidia-smi".
TWO THINGS THE API TAUGHT US
1. `localSSDCount` is rejected with machineFamily. GKE's webhook:
compute-class "io" doesn't support local ssd without machine type or GPU
config
Local SSD attachment is a per-machine-type property, so io must name explicit
machineTypes. The cost is a narrower fallback list than general-purpose gets.
Found by `kubectl apply --dry-run=server`, not by reading docs.
2. `cluster_autoscaling.gpu_resources` was an empty list, deferred when the GPU
class did not yet exist. It is now REQUIRED: NAP will not create a node with
an accelerator that has no resourceLimits entry, so leaving it empty caps GPU
autoscaling at zero -- the class applies cleanly, pods stay Pending, and
nothing says why. nvidia-l4, maximum 2.
COST POSTURE UNCHANGED
Both classes are spot-only with `whenUnsatisfiable: DoNotScaleUp`. That matters
most for gpu-l4: spot GPUs are scarcer than spot CPU, so the class will
sometimes provision nothing -- which is the intended behaviour, because an L4
silently falling back to on-demand is the single most expensive accident
available in this repository.
Verified: `kubectl apply --dry-run=server` accepts both against the live GKE
API; ./scripts/validate-manifests.sh -> Valid: 1191, Invalid: 0, Skipped: 0,
all gates passed; tofu validate and tofu fmt clean.
NOT verified: neither class has actually provisioned a node. general-purpose was
exercised end to end (criteria 12, 14, 17); io and gpu-l4 are schema-valid and
follow a proven pattern, which is not the same as measured. Criterion 15 in
particular needs a real GPU pod.
Every criterion now carries its outcome and the evidence for it, rather than
leaving the design as a list of intentions.
Four PASS, measured on gcp-mycluster-0:
12 taint applied at registration and cleared by Cilium
14 e2-highcpu-4, spot=true, cos_containerd -- NAP picked highcpu over
standard unprompted, cheaper per vCPU
16 a 64-vCPU pod against a 32-vCPU ceiling held Pending 4 minutes, node count
never moved, autoscaler logged NotTriggerScaleUp
17 all four auto-created nodes reaped after the probe was deleted
One PARTIAL (13), one BLOCKED (15).
13 holds as worded -- nothing recorded a MISSING CNI -- but the toleration that
criterion 12 forces means tolerating pods can land before the Cilium agent is up
and log a transient `cilium-cni failed (add) ... EOF`. It self-heals, and
kube-system/metrics-server hits the same on any fresh node. Recording that is
more useful than a green tick, because the two failure modes read almost
identically in logs and only one is benign.
15 is BLOCKED, not failed, and the distinction is the point:
GPUS_ALL_REGIONS on this project is 0, so no GPU node can exist at any price.
Worth writing down loudly because the obvious diagnostic misleads -- the
REGIONAL NVIDIA_L4_GPUS quota reads 1, which looks fine and would send the next
person away to "retry when spot frees up". The spot attempt fails with
`GCE out of resources` and only the on-demand attempt says `GCE quota exceeded`.
Running both is what distinguished capacity from quota.
Everything up to the GPU is proven: the class is selected and NAP resolves it to
a g2-standard-4-gpu1 pool and attempts creation. That also retroactively
confirms cluster_autoscaling.gpu_resources is load-bearing -- left empty, no
scale-up is attempted at all and the symptom is silence.
The on-demand probe cost nothing: the node was never created, and the throwaway
ComputeClass and pod were deleted immediately.
Also adds the quota blocker to the resume plan, where someone rebuilding will
look, with the explicit warning not to diagnose it from the regional quota.
Verified: ./scripts/validate-links.sh -> all relative links resolve.
Contributor
🔍 Rendered manifest diff — this PR vs
|
Six findings from the review, all applied. The first is a flat contradiction and the second is a failure this session already fixed once elsewhere. 1. general-purpose.yaml opened with "the FIRST and, for now, the ONLY one" and "Deliberately one class, not three". Both false in this branch -- io and gpu-l4 sit beside it, and kustomization.yaml lists all three. That was true when written and never updated when the other two landed. The first thing a reader saw was wrong, so the paragraph is gone; kustomization.yaml already records the general-purpose-proven-first history, once and better. 2. The taint story was told FOUR times in general-purpose.yaml -- why it matters, the measurement, why it is there, the trade -- and a fifth time in ADR-0006, near-verbatim down to the `cilium-cni ... EOF` line. That is exactly the cilium.yaml failure corrected earlier in this branch's history: overlapping accounts of one decision, where the next amendment updates some and silently strands the rest. Now: the file states the REQUIREMENT (the toleration snippet, ready to copy, plus one sentence that without it nothing scales up), and the ADR explains it. One account, one place to amend. io and gpu-l4 keep only their own class-specific taint sentence -- ogenki/io and nvidia.com/gpu are genuinely local facts -- and cross-reference the rest. 3. imageType had three different treatments across three files: five comment lines, three, and none. Now one identical line in each, pointing at the image_type note in main.tf where the reasoning lives. 4. The criterion-16 ceiling rationale was stated in main.tf and again on the variables in variables.tf. Kept on the variables, where the default sits. 5. gen-catalog.sh listed "Three sources" with the new entry numbered 4 and inserted between 2 and 3. Now four sources, in order. 6. infrastructure.yaml argued carefully about the dependsOn it OMITS (crds) and left the one it keeps unexplained. Now says why: currently inert, since a ComputeClass is cluster-scoped and this path holds nothing namespaced, but kept because the first namespaced resource added here would otherwise race its namespace -- a failure that presents as a dependency cascade rather than as a missing namespace. Deliberately NOT changed: the taint declarations are duplicated across all three classes and stay that way. The review's own recommendation, and it is right -- the taints are what a reader opens the file to learn, and a kustomize patch injecting them would make io.yaml look like it needs one toleration when it needs two. Roughly ten shared YAML lines is below where indirection pays. Verified: ./scripts/validate-manifests.sh -> Valid: 1191, Invalid: 0, Skipped: 0, all gates passed; tofu validate and tofu fmt clean; bash -n clean on gen-catalog.sh; the API still accepts all three classes.
Upgrades io from 'schema-valid, unexercised' to verified. It provisioned an n2-standard-4 spot node with a 375 GiB Local SSD attached -- 368 GiB ephemeral storage on a 50 GB boot disk is the giveaway -- and the probe pod reached Running. Tested because the GPU result made a second quota blocker look likely: PREEMPTIBLE_LOCAL_SSD_GB reads 0 in europe-west4, and io is spot + localSSDCount 1, which appeared to be exactly the gpu-l4 situation again. It is not. GKE-managed Local SSD on spot nodes provisions regardless of that metric. The prediction was wrong, and the false alarm is recorded in the design so the next person does not re-derive it and 'fix' a class that works -- the regional LOCAL_SSD_TOTAL_GB reads effectively unlimited, so the two quotas together are genuinely confusing. Per-class status is now explicit in the design: general-purpose and io VERIFIED, gpu-l4 proven up to provisioning and blocked on GPUS_ALL_REGIONS: 0. Verified: ./scripts/validate-links.sh -> all relative links resolve.
…orted on G2 Correctness review found a blocker that no gate here could have caught, plus two smaller real issues. One of its suspicions is refuted by measurement. BLOCKER: gpu-l4 was unprovisionable. cluster_autoscaling sets disk_type = "pd-standard" as the boot disk for EVERY auto-created pool, and gpu-l4 overrode nothing. The G2 series does not support pd-standard -- Google's PD matrix allows only pd-ssd, pd-balanced and hyperdisk there, and the ComputeClass CRD says so itself: "available boot disk types depend on the machine family". The failure mode is the bad one: NAP either filters g2 out as unsatisfiable or the pool create fails on disk type, and with whenUnsatisfiable: DoNotScaleUp no event names the disk. A GPU pod just stays Pending -- exactly the silent mode the gpu_resources comment warns about. `kubectl apply --dry-run=server` provably cannot catch it: it never reaches VM creation. Nor could tofu validate, or the manifest gate, or the live test, because GPUS_ALL_REGIONS: 0 stopped us one step earlier. Two independent blockers were stacked and only the outer one was visible. Fixed with `storage.bootDiskType: pd-balanced` on the g2 priority rather than by changing the cluster-wide default: e2 and n2 both support pd-standard and are the common case, so only GPU nodes pay for the dearer disk. CONFIRMED: the autoscaling ceiling is cluster-wide, not per-class. GKE counts manually created pools toward max_cpu_cores. With the static pool at its max of 3 x e2-standard-4 (12 vCPU), only 20 of the 32 vCPU remain for all three classes combined -- so gpu_resources maximum 2 (8 vCPU) is reachable only while general-purpose and io stay under 12. A GPU scale-up can be starved by CPU scale-up, and the events do not distinguish the two. Documented on the variable. SUSPECTED, recorded not fixed: GKE applies its own nvidia.com/gpu=present:NoSchedule to GPU nodes, and Kubernetes forbids two taints sharing key+effect. Either GKE's value wins -- making our declared "true" inaccurate -- or the pool create errors. The toleration this file recommends uses operator: Exists, which survives both, so the block now says so explicitly and asks whoever sees the first GPU node to confirm and correct it. Guessing would be worse than flagging. REFUTED by measurement: the review suspected localSSDCount attaches raw NVMe block rather than pod ephemeral storage, which would have broken the AWS parity claim. The io node provisioned earlier reported 385926528Ki (~368 GiB) of ephemeral-storage on a 50 GB boot disk -- that is the Local SSD and nothing else. Recorded in io.yaml so the question is not reopened. Verified: ./scripts/validate-manifests.sh -> Valid: 1191, Invalid: 0, Skipped: 0; the API accepts the modified gpu-l4; tofu validate and fmt clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Design slice 4: GKE node auto-provisioning (ADR-0006) and the three ComputeClasses.
Built and measured on a live cluster. Four criteria PASS, one partial, one
blocked on a GCP quota — each annotated with its evidence in the design doc
rather than left as an intention.
Two halves, both required
OpenTofu —
cluster_autoscalingon the GKE module. Without it a ComputeClasswith
nodePoolAutoCreationstill schedules, but only onto pools that alreadyexist: nothing is ever provisioned and the slice's premise goes untested.
Manifests — three classes under a new
infrastructure/gcp-mycluster-0/,mirroring the AWS Karpenter NodePools:
general-purposedefault-nodepoole2→n2, no on-demand fallbackioio-nodepoologenki/ioidentical to AWSgpu-l4gpu-l4-nodepoolnvidia.com/gpuidentical to AWSTaint keys match AWS deliberately: they're platform conventions, not cloud
details, so a workload's tolerations are the one part of its manifest that
doesn't change between clouds.
The finding the slice existed to produce
Criterion 12 passes — an auto-created node carries
node.cilium.io/agent-not-readyand Cilium clears it. But testing it surfaced something the design did not
anticipate:
Every workload targeting a ComputeClass must tolerate that taint, or nothing
scales up at all. The autoscaler simulates scheduling against a node that
will carry it, judges an untolerating pod unplaceable, and never provisions.
The symptom is silence: three Pending pods for ten minutes with no
TriggeredScaleUpevent of any kind. Adding only the toleration produced fournodes and all pods Running.
That's a real divergence from AWS, where the taint is invisible to workloads
because static pools carry it and Karpenter provisions against
NodePoolrequirements rather than simulating a tainted node.
Decision (yours): keep the taint, require the toleration. It still gates
every pod that doesn't opt in and keeps auto-created nodes behaving like static
ones. Recorded in ADR-0006 as a Negative consequence — and its Positive list,
which claimed the taint is "handled declaratively at the pool level", is amended,
because that is true but incomplete in exactly the way that cost ten minutes of
silent Pending.
Criterion 15 is blocked, not failed
GPUS_ALL_REGIONSon this project is 0, so no GPU node can be created at anyprice.
Worth flagging loudly because the obvious diagnostic misleads: the regional
NVIDIA_L4_GPUSquota reads 1, which looks fine. The spot attempt fails withGCE out of resources— suggesting "retry when capacity frees up" — and only anon-demand attempt reveals
GCE quota exceeded. Running both is whatdistinguished capacity from quota.
Everything up to the GPU is proven: the class is selected, NAP resolves it to a
g2-standard-4-gpu1pool and attempts creation. That also retroactively confirmscluster_autoscaling.gpu_resourcesis load-bearing — left empty (as it was untilthis PR), no scale-up is attempted at all and the symptom is silence.
Closing it needs a quota request to Google. Recorded in the resume plan.
Two things the API taught us
localSSDCountis rejected withmachineFamily: "doesn't support local ssdwithout machine type or GPU config". Local SSD attachment is per-machine-type,
so
ionames explicit types. Found via--dry-run=server.ComputeClassneeded a fourth schema-catalog source. The repo runsskipMissingSchemas: falseby design, and it's in no public catalog — unlikethe AI Gateway and Karpenter CRDs which are rendered from pinned charts, GKE
installs this one and publishes no chart. Vendored under
scripts/flux-schema/vendored-crds/with its GKE component version recorded.Deliberately not under
crds/, which Flux applies: it's GKE addon-managedand applying it would fight GKE.
Cost
All three classes are spot-only with
whenUnsatisfiable: DoNotScaleUp, soexhausted spot leaves pods Pending rather than silently going on-demand — which
matters most for L4s.
cluster_autoscalingwas also sitting on module defaults of 100 GBpd-standard, twice the static pool's 50 GB, purely because the field wasunset. Now 50 GB and the cheapest disk type.
Verification
./scripts/validate-manifests.sh→Valid: 1191, Invalid: 0, Skipped: 0kubectl apply --dry-run=serveraccepts all three against the live GKE API./scripts/validate-links.sh→ all links resolvetofu validate,tofu fmt,shellcheck -x -S warningcleanback to its 2 static nodes, no GPU node was ever created