design: component workload overrides

[Philip Lombardi (plombardi89)]

Adds designs/component-workload-overrides.md: a mechanism for users to customize the Deployments and DaemonSets that unbounded-operator generates and reconciles.

Design document only. No code, no CRD changes, no make generate.

This description is self-contained and supersedes the earlier point-by-point review reply, which has been removed so there is one source of truth.

---

The problem

unbounded-operator generates and reconciles the workloads for five components: net, machina, gantry (cluster singletons) and metalman, storage (per-Site). A user's entire influence over their shape today is:

• spec.components.<c>.enabled (api/machina/v1alpha3/site_types.go:147)
• spec.components.metalman.replicas (:167) and dhcpAutoInterface (:161)
• The operator-wide --image-registry flag (cmd/unbounded-operator/main.go:76)

The requirements collected from users and the team are considerably larger: resources, tolerations / nodeSelector / affinity, sidecars and volumes, environment variables, command arguments, imagePullSecrets, labels / annotations / priorityClassName, and container images.

There is no workaround, supported or otherwise. The operator applies with server-side apply and ForceOwnership (internal/operator/component/env.go:242), so a kubectl edit is reverted on the next reconcile and a GitOps controller managing the same object fights the operator indefinitely. Customization has to be an input to the operator or it cannot exist.

This reverses a documented stance. site_types.go:141-143 says components are deliberately not configurable, and architecture.md:189 says "no Helm or Kustomize". §1 of the doc quotes both; PR 5 amends them.

The solution space

	Mechanism	Expressiveness	Mechanism reach	Validation	Cost
A	Kustomize as a library	Unlimited	Any object, any GVK	None practical	High
B	Typed override struct on the CRD	Closed set of fields	Chosen fields	Full OpenAPI	Low, permanent field creep
C	Strategic merge over allowlisted paths on operator-emitted workloads	Allowlisted paths, open values	One workload object, GVK pinned	Allowlist + re-stamp + apply-time GVK assertion	Low
D	SSA field disownership	"Let X own this" only	Chosen paths	N/A	Very low
E	User-supplied MutatingAdmissionPolicy	High	Any object admission sees	CEL type-checked	Zero code

C is proposed. Every requirement above is a field inside a Deployment or DaemonSet the operator already emits. None needs object creation, renaming, or deletion.

Why not kustomize (§5.1)

This was the team's initial suggestion, so the doc argues it at length rather than dismissing it. Five reasons, of which two are load-bearing:

Unbounded transformation with no pruning. A kustomization can rename, delete, add, and re-kind resources. The operator never prunes; singletons are deliberately retained even when no Site enables them (machina.go:57-69). An overlay that renames a DaemonSet orphans the original permanently and the operator recreates it next pass.

GVK containment is impossible by construction. This is subtler than "kustomize grants more privilege", which is not actually true (see the security caveat below). A patch surface can be pinned to apps/v1 Deployment and DaemonSet by validation, by re-stamping, and by an assertion immediately before apply. An overlay engine cannot be, because selecting group, version and kind is precisely what it exists to do; constraining it to two GVKs removes the reason to adopt it. The difference is between an attacker who must pivot through a compromised node and one who writes a ClusterRoleBinding directly using the escalate and bind verbs the operator holds (deploy/unbounded-operator/02-rbac.yaml.tmpl:60-66).

Also: adopting it freezes the deploy/*/rendered/ layout and every object name in it as public API; kustomize consumes a filesystem, which maps badly onto a cluster object; failed builds produce a string with no per-component attribution; and it promotes sigs.k8s.io/kustomize/api from transitive (go.mod:336) to direct.

Why not a typed struct (§5.2)

Sidecars, volumes, env, args and images cannot be covered by a closed set of fields in any meaningful sense. Projects that start there add a free-form pod template later and carry two overlapping surfaces. This is not contradicted by the allowlist, which is a closed set of paths rather than of fields: values stay open, and users can address containers and volumes the operator never enumerated, which a Go field plus CRD schema per knob cannot.

What is proposed

Storage. A user-owned ConfigMap unbounded-component-overrides in the operator namespace. The operator only reads it; never creates, seeds, or writes it. Chosen over Site.spec because the net/machina/gantry singletons resolve enablement as "any Site enables it" (machina.go:47-55), so a per-Site override field is ambiguous when Sites disagree. A cluster-scoped object dissolves that and avoids a v1beta1 conversion obligation. Documents are versioned by a required apiVersion.

Merge. k8s.io/apimachinery/pkg/util/strategicpatch with NewPatchMetaFromStruct. No new dependency. Merge keys verified against k8s.io/api@v0.36.3. The patch targets the whole workload object, so spec.replicas and workload metadata are reachable through the same field as spec.template.spec.*.

Allowlist, not denylist (§8). Unenumerated paths are rejected, so fields added by future Kubernetes versions are denied by default. Protected and re-stamped: GVK, name, namespace, ownerReferences, finalizers, selector, selector-referenced template labels, serviceAccountName, host namespaces, the unbounded-cloud.io/ prefix, and operator-declared volumes. Also rejected: any $-prefixed key at any depth, and explicit nulls.

Additive scheduling (§8.3). NodeSelectorTerms carries no patchMergeKey (k8s.io/api/core/v1/types.go:3778), so a raw patch supplying nodeSelectorTerms replaces the mandatory Site affinity that metalman and storage depend on (metalman.go:170, storage.go:261-268), silently allowing two Sites' workloads onto the same nodes. Scheduling constraints are therefore ANDed into operator terms rather than patched. Same for nodeSelector and tolerations.

Failure semantics (§9). Validation is atomic across the whole ConfigMap before any write. On failure the operator retains last-known-good rather than reverting, because a single typo would otherwise strip resources and scheduling constraints from every component at once. Failure scope is tabulated; override-hash-desired and override-hash-applied make divergence observable.

Application point (§10). Inside Env.ApplyObject, the only place the four YAML-driven components and metalman's typed path converge, so both get identical semantics with no component file changes.

CLI (§12). kubectl unbounded overrides list, validate, and status.

---

The security caveat

Write access to the overrides ConfigMap is equivalent to root on every node in every affected Site, and therefore to cluster-admin. This is a property of the mechanism, not a defect in it, and §4 states it plainly rather than implying a boundary that does not exist.

Restricting privileged containers and host mounts does not change this, because the workloads are already maximally privileged:

Workload	Privilege
unbounded-net-node	hostNetwork, hostPID (deploy/net/node/03-daemonset.yaml.tmpl:32-33), privileged: true on two containers (:53, :108), four hostPath mounts (:125-137)
unbounded-storage-supervisor	privileged: true (04-daemonset.yaml.tmpl:64,100), three hostPath mounts (:103-111)
metalman	HostNetwork: true (metalman.go:165)

Rejecting privileged: true is a no-op against containers that are already privileged. Rejecting hostPath is a no-op when the host root is already mounted. Meanwhile an image change, an args change, an LD_PRELOAD env injection, or a sidecar inheriting pod-level hostPID and hostNetwork are each arbitrary root execution on every node.

A field allowlist bounds this only if image, args, env, sidecars and volumes are all excluded, which is option B and removes five of the eight requirements. The requirement set and containment are in direct conflict, and this design resolves it by accepting the privilege level explicitly.

Consequences:

• The allowlist is an integrity control, not a security control (§4.2). It stops an authorized operator from accidentally severing the operator's ability to reconcile. GVK and serviceAccountName are the two exceptions and are genuine security controls, because they escape the workload rather than damage it.
• Access must be restricted like RBAC (§4.3). Cluster administrators only; audited; not part of namespace-wide ConfigMap grants.
• There is a residual gap (§4.4). machina-controller (deploy/machina/02-rbac.yaml.tmpl:15-17), metalman-controller (06-metalman-rbac.yaml.tmpl:59,132) and unbounded-net-controller (net/controller/02-rbac.yaml.tmpl:170,173,216) hold namespace-wide ConfigMap write with no resourceNames. A compromised component can write the overrides ConfigMap. PR H1 narrows these, but RBAC cannot scope create by resourceNames, so a component retaining create can still seed the object when absent. Only a dedicated resource type closes this. That tradeoff is open question 1, not a settled decision.

Other caveats

• Args replacement. args and command carry no patchMergeKey, so a patch replaces them wholesale and drops operator-injected flags. metalman makes this concrete: its args begin with the serve-pxe subcommand (metalman.go:108), so a replacing patch stops the container starting. extraArgs exists for the append case.
• Zero-available windows. net-controller and metalman use maxSurge: 0 with maxUnavailable: 1 because both are host-networked. A bad override yields a window with no available replica, and since Ready=True only means the apply succeeded, the Site will not report it.
• Observability does not exist yet. Site declares no condition printer columns (site_types.go:25-33) and SiteReconciler has no event recorder; only LegacyReaper does. Both are implementation work in PR 3, not assumptions.
• Validation timing. The API server accepts any ConfigMap, so errors surface in reconcile. overrides validate and last-known-good make the window inert rather than destructive.
• Last-known-good is in-memory. A restart holding an invalid document applies no overrides. Stated rather than solved; open question 4.
• Revert is scoped. Only fields the operator currently declares, on objects it currently emits. Admission mutations, competing field managers, and the absence of pruning can all retain state.
• Pinned images survive upgrades indefinitely and are the likeliest cause of an install behaving unlike its reported version.

---

Review response

Changes Requested was raised against the first revision. All ten findings are addressed across 112fa7c, 5526b4d and 94cc276. Nine accepted; finding 1's remedy partially declined with reasoning below.

Sections were renumbered by two insertions:

Was	Now
-	§4 Security model (new)
§4 Alternatives	§5
§5 Format	§6
§6 Merge semantics	§7
§7 Ordering and invariants	§8 Permitted and protected fields (rewritten)
-	§9 Failure and update semantics (new)
§8 Wiring	§10
§9 Drift visibility	§11 (+ observability)
§10 CLI	§12
§11 Operational notes	§13
§12 Implementation plan	§14
§13 Testing	§15
§14 Prior art	§16
§15 Open questions	§17

#	Finding	Disposition
1	Override writers gain cluster-admin execution	Accepted; remedy partially declined. See below.
2	Patch can change resource GVK	Accepted, and understated. The operator holds escalate and bind on clusterrolebindings, so this was a direct path to cluster-admin, not merely "another resource". Fixed with three independent layers (§8.2, §8.4): validation rejection, post-merge re-stamp, and an apply-time apps/v1 assertion plus a defensive check in ApplyObject. Recorded as constraint §3.7.
3	Per-Site workloads can be retargeted	Accepted. Worse than described: NodeSelectorTerms has no patchMergeKey, so any patch supplying it replaces Site affinity. §8.3 makes scheduling additive; §15 adds a regression test that two Sites cannot be co-scheduled through any permitted override.
4	Denylist does not preserve invariants	Accepted. Replaced with an allowlist that fails closed. Added serviceAccountName, host namespaces, finalizers, reserved prefix, operator-declared volumes. Rejection extended to any $-prefixed key at any depth and explicit nulls. Also picked up retainKeys on Volumes (types.go:4145), where a partial patch drops sibling fields. Did not adopt the PodTemplate-plus-workload-fields split, because spec.replicas, spec.strategy and workload metadata are all in the requirements and would need a parallel surface.
5	Invalid-update behavior undefined	Accepted. New §9: atomic validation, last-known-good retention, tabulated failure scope, desired vs applied hashes. In-memory limit stated (open question 4).
6	Watch can permanently miss per-Site updates	Accepted. Confirmed: watch.go:44-49 drops the fan-out on List failure with no retry, and reconciler.go:179 means the singleton pass never runs Site components. §10 moves fan-out into Reconcile where a failed List returns an error and controller-runtime retries. §15 adds a regression test that must fail against the original wiring.
7	Compatibility contradicts upgrade behavior	Accepted. §2 splits syntactic compatibility (guaranteed within an apiVersion) from target resolution against release-specific container and volume names (not guaranteed). An absent extraArgs container is reclassified as an object-scoped resolution failure.
8	CLI rendering not authoritative under skew	Accepted; feature dropped. All three sub-points correct, including that ensureConfig writes before render (net.go:64, gantry.go:109), so a pure render would either omit the config hash or perform writes. overrides diff removed (§12.1) and replaced with overrides status. This removed the five-component render/apply refactor from the plan.
9	Observability claims do not match plumbing	Accepted. All three verified wrong. §11 now opens with what does not exist and adds a printer column and an EventRecorder as explicit work.
10	Revert guarantees too broad	Accepted. §2 narrowed; §15 moves the revert test onto envtest against real managedFields, plus a test documenting that a competing manager's field survives.

On finding 1

The finding is correct and the RBAC over-grants are real. Where this differs is on the remedy. Restricting privileged containers and host mounts does not work, for the reasons in the security caveat above: the workloads are already privileged, so those restrictions are no-ops, while image, args, env and sidecar changes remain root execution.

Rather than imply containment that cannot exist, §4 accepts the privilege level, documents it as cluster-admin-equivalent, gives required RBAC posture, and discloses the residual create gap that PR H1 does not close. The dedicated-resource alternative is not dismissed; it is recorded as open question 1 with the create limitation spelled out, so it can be re-argued on the merits. If the reviewer still favours it, that is the place to push and it will not be defended on principle.

---

Implementation sequence (§14)

PR	Scope	Depends on
1	This design document	-
2	Override engine: load, atomic validation, allowlist, resolution, composition, conflict detection, merge, additive scheduling, re-stamp, GVK lockdown, last-known-good, hashes. Pure functions plus unit tests.	1
3	Wiring: Env.ForComponent, ApplyObject gate and kind assertion, retryable fan-out, ReconciledWithOverrides, EventRecorder, Site Overrides printer column.	2
4	kubectl unbounded overrides list, validate, status	3
5	Docs, including architecture.md:189, site_types.go:141-143, cli.md, and the access-control guidance from §4.3	2, 3, 4
H1	RBAC hardening, narrowing the namespace-wide ConfigMap grants. Independent, not blocking and not blocked. Worth doing on its own merits.	-

Only the printer column requires make generate. No API version churn.

Open questions (§17)

Please reply against the numbers. 1, 2 and 4 most need a second opinion.

1. Dedicated resource type instead of a ConfigMap. Is the smaller API surface worth the residual create gap?
2. Admission-time validation. Would a ValidatingAdmissionPolicy covering schema, apiVersion, protected paths, $ directives and nulls be worth the extra installed object, given it cannot do resolution?
3. Operator-side dry-run, now that client-side diff is rejected.
4. Last-known-good across restarts. Is applying no overrides after a restart with an invalid document acceptable?
5. spec.replicas vs the typed spec.components.metalman.replicas. Which wins?
6. siteSelector deferred in favour of a sites name list. Agree?
7. ServiceAccount annotations. serviceAccountName is protected, but workload identity normally requires annotating the SA, and the operator reverts those under ForceOwnership. A real gap neither this nor the existing escape hatch covers.
8. SSA field disownership as a complement for the "let VPA own resources" case.
9. Reserved container names for sidecars, or is a documented convention enough?

Review notes

• designs/ is not published to the docs site; docs.yaml only fires on docs/**.
• ci.yaml has no path filter, so the full matrix runs on this docs-only PR.
• Status is Draft for team review. It flips to Accepted in a final commit here once the open questions resolve, then squash merge.