feat(gcp): scoped Crossplane IAM identity for slice 5 - #1823
Merged
Conversation
Restores the binding removed in #1818 -- which had no consumer then and does now -- with least privilege rather than breadth. BEFORE: roles/editor, project-wide. Thousands of permissions across every service, granted to a principal nothing created. AFTER: one binding, roles/resourcemanager.projectIamAdmin, conditioned so it may grant only an allowlist: api.getAttribute('iam.googleapis.com/modifiedGrantsByRole', []) .hasOnly(['roles/dns.admin']) WHY THE CONDITION IS THE POINT projectIamAdmin alone would be a large improvement on editor and STILL a privilege-escalation path: setIamPolicy can grant any role to any principal, including granting Crossplane itself roles/owner. The condition closes that. It is also the honest GCP analogue of the AWS side's `xplane-*` scoping. AWS restricts Crossplane by resource NAME; GCP cannot for project IAM, because the resource IS the project -- so it restricts by grantable ROLE instead. Same intent, different axis, and worth stating because the earlier conclusion in this repo was that GCP simply had no equivalent. TWO THINGS MEASURED, NOT READ 1. GCP IAM conditions run a RESTRICTED CEL dialect. The first attempt used `.all(r, r in [...] || r.startsWith(...))` to allow both predefined roles and xplane-prefixed custom ones. GCP rejected it at apply: Condition expression compilation failed: undeclared reference to '@not_strictly_false' The `.all()` macro is unavailable; `hasOnly()` is the supported form and matches exact strings only. tofu validate passes on the broken version -- only the API knows. 2. That forced a tighter design, which is the better one. Exact names mean dynamically-named custom roles cannot be allowlisted, so roles/iam.roleAdmin is NOT granted at all: GCPWorkloadIdentity's optional customRole.permissions is deferred rather than half-enabled. Slice 5's actual need -- criterion 21, external-dns records and cert-manager's DNS-01 challenge -- is served entirely by roles/dns.admin. The file records what re-enabling customRole would require, and that widening the condition is not it. Both traps from the removed version are carried forward: the NUMBER/ID split in the principal string (reversed, the API accepts it and it never matches), and the missing graph edge to module.gke that fails only a FRESH apply. Verified on the live project: `gcloud projects get-iam-policy` shows exactly one crossplane binding, with the condition attached and the member in the precise principal://.../ns/crossplane-system/sa/crossplane form design criterion 19 requires. No roles/iam.roleAdmin binding exists.
The binding committed in 33a02f6 could never have matched. It named `ns/crossplane-system/sa/crossplane`, but Crossplane core never talks to GCP -- the PROVIDER pod makes the cloud API calls, under its own ServiceAccount. The AWS side has said so all along: opentofu/aws/eks/init/iam.tf:56-57 binds its Pod Identity to crossplane-system/provider-aws, not to crossplane. I found this by checking what AWS actually binds while writing the GCP provider manifests, not from any error -- and there would not have been a useful one. This is TRAP 1 from the same file, in its other form. A wrong subject is ACCEPTED by the API and simply never matches: no validation failure, no Crossplane error at install, just permission denials at first use that point at the workload rather than at the binding. The name is now tied to its source in a comment: it must equal the serviceAccountTemplate in the DeploymentRuntimeConfig that slice 5's provider tree will carry. The two are set in different repositories' worth of context and nothing checks that they agree. Verified on the live project: the stale sa/crossplane binding is destroyed and exactly one remains -- roles/resourcemanager.projectIamAdmin ns/crossplane-system/sa/provider-gcp condition: modifiedGrantsByRole.hasOnly(['roles/dns.admin']) which is the precise principal:// form design criterion 19 requires.
Contributor
🔍 Rendered manifest diff — this PR vs
|
Records a decision that workstream 11 would otherwise have had to make under
pressure, and separates two workstreams that are easy to conflate.
THE SPLIT, which was the first thing to get straight:
10 PUBLIC certs -- cert-manager clouddns DNS-01 against Let's Encrypt.
Depends on slice 5.
11 PRIVATE certs -- OpenBao's own PKI, the GCP counterpart to what
bao.priv.aws.ogenki.io serves. Depends only on
workstream 1, so it is NOT blocked by slice 5.
Only 11 needs OpenBao running. Both rows now cross-reference the new section
instead of leaving the reader to infer which is which.
DECISION: two OpenBaos, two roots.
Rejected -- a shared OpenBao reached over the tailnet: zero new infrastructure
and one trust anchor, but it makes GCP certificate issuance hard-depend on AWS
and on the tailnet, in a platform whose stated point is that each cloud stands
alone. The design already flags that same coupling as undesirable for the Flux
GitHub App secret.
Rejected -- two OpenBaos sharing one root: one trust anchor AND independent
operation, which looks like the best of both. It is not, operationally. It needs
either the root PRIVATE KEY copied into GCP Secret Manager, doubling exposure of
the most sensitive material the platform holds, or GCP's intermediate
cross-signed at bootstrap -- a manual ceremony on EVERY REBUILD of a platform
whose lifecycle is build-validate-destroy.
The deciding evidence is from this session rather than from theory. The private
domain rename forced a new OpenBao server certificate, and re-issuing it under
the existing chain turned out to be impossible: the intermediate that had signed
it had no private key stored anywhere, because OpenBao issued it and the key
never left OpenBao. A fresh CA was the only way forward.
That is precisely the failure option C institutionalises, and it fires at
rebuild time -- the worst moment. Two independent roots cost one extra trust
anchor and delete the whole class.
Consequences carried into workstream 11 rather than left implicit: tailnet
clients must trust both roots; GCP needs its own Secret Manager entries for the
root token and cert-manager AppRole, mirroring AWS Secrets Manager and following
the pattern flux-github-app already set; Cloud KMS auto-unseal is what makes an
unattended rebuild possible; and the per-cloud private domains from ADR-0017
make the split clean, since no name resolves to either CA ambiguously.
Nothing here reaches application manifests. Workloads request a cert-manager
Certificate, which is already cloud-neutral -- the issuer differs per cloud, the
developer-facing API does not, which is ADR-0007's split by audience.
Also added to the resume plan, since workstream 11 can start independently and
whoever picks it up will look there first.
Verified: ./scripts/validate-links.sh -> all relative links resolve.
Seven findings applied. The first was a wording error that turned out to be covering a hole in the plan. THE CONTRADICTION, AND WHAT WAS BEHIND IT The new PKI section said workstream 10 issues PUBLIC certificates "for names under priv.gcp.ogenki.io" -- while the same section, and ADR-0017, say priv.gcp.ogenki.io is the PRIVATE zone and public stays cloud.ogenki.io. A section written to stop workstreams 10 and 11 being conflated opened by conflating them. Correcting the domain exposed the actual problem: WORKSTREAM 10's DNS-01 HAS NOTHING TO SOLVE AGAINST. opentofu/gcp/network/dns.tf creates a PRIVATE Cloud DNS zone and nothing else, and Let's Encrypt must resolve the _acme-challenge TXT record publicly. cloud.ogenki.io is a Route53 zone this repository does not even manage -- it appears only as a data lookup. So "cert-manager clouddns DNS-01" was never a complete plan, and asserting the wrong domain hid that. Three ways out are now recorded, none chosen: solve DNS-01 against Route53 from GCP (works today, reintroduces the cross-cloud dependency option A was rejected for); delegate a public subdomain to a new public Cloud DNS zone; or serve no public certificates from GCP and keep public ingress on AWS, which is coherent while GCP has no public endpoints. Worth settling before workstream 10 starts rather than during it. Workstream 11 is unaffected. THE DEAD REFERENCE iam.tf pointed at "the DeploymentRuntimeConfig's serviceAccountTemplate in infrastructure/gcp-mycluster-0/crossplane/providers/" in the present tense. That directory does not exist, and `provider-gcp` appears nowhere else in the repo -- a reader greps, finds nothing, and cannot tell whether the binding or the path is wrong. Restated as what it is: an OBLIGATION on slice 5, with the warning that nothing checks the two agree. DUPLICATION, the pattern that keeps recurring here Four restatements removed: a verbatim re-telling of TRAP 1 twelve lines below the original; a forward-pointer to a block in the same file; a second copy of the dns.admin rationale; and a re-summary of the 2026-08-23 OpenBao correction that already sits earlier in the same document. The resume plan also re-told the design's whole rationale after naming it as the source -- now reduced to the sequencing facts that are the plan's actual job, with a link for the reasoning. The measured-behaviour comments are untouched: the CEL .all() rejection, the NUMBER/ID split, the provider-SA trap, additive-vs-authoritative. Those are the file's value; the findings were about copies of them. THE REVIEWER'S CLOSING CHALLENGE, ACCEPTED This binding again lands ahead of its consumer -- exactly the situation #1818 removed the old one over. The header no longer glosses that. It states the caveat, why it is acceptable here where roles/editor was not (blast radius is one grantable role, not owner; and the provider cannot authenticate without it, so it must precede the deployment), and instructs removing it again if slice 5 stalls. Verified: tofu validate and tofu fmt clean; ./scripts/validate-links.sh -> all relative links resolve.
Security review of the slice 5 Crossplane binding found that roles/dns.admin carries far more than external-dns and cert-manager need: - dns.managedZones.delete contradicts the platform constitution's "no deletion permissions for stateful services (S3, IAM, Route53)". The AWS side honours that rule; this did not. - The dns.responsePolicies.* / dns.policies.* family lets a compromised provider-gcp bind a response policy to the cluster VPC, overriding metadata.google.internal or *.googleapis.com to redirect in-cluster traffic and harvest credentials — invisible to external-dns. Replace it with a pre-created xplane_dns_editor custom role holding only record-set management, transactional changes, and read-only zone lookup. Pre-creating it in OpenTofu also gives it a deterministic name, which is what makes it allowlistable at all: the IAM condition matches exact names via hasOnly, so a composition-rendered role never could be. Also grant a read-only xplane_role_reader. projectIamAdmin's 9 permissions do not include iam.roles.get, and referencing a custom role in setIamPolicy can require reading it — a constraint the predefined-role draft would never have hit. Whether GCP enforces it here is unverified (needs a live provider pod), so it is granted pre-emptively: the downside is asymmetric and reading role definitions confers nothing. Record three gaps the condition cannot close rather than leaving them assumed safe: it gates 1 of the role's 9 permissions (modifiedGrantsByRole is undefined for non-setIamPolicy verbs, making hasOnly vacuously true — latent until the org adopts Principal Access Boundaries); it constrains the role and never the member; and it is scoped to the project-wide workload identity pool, so every future cluster in the project inherits it. The last was documented in #1818 and lost in the rewrite. Confirmed while reviewing: modifiedGrantsByRole covers revocations as well as grants, so the condition also stops Crossplane removing bindings it did not create, including break-glass human access. Update the design's example claim, which asked for roles/dns.admin and would now be refused by the condition, and mark customRole.permissions unusable.
Smana
added a commit
that referenced
this pull request
Aug 24, 2026
* feat(gcp): Crossplane controller, provider and config tree Slice 5's cluster-side plumbing: the GKE cluster can now run Crossplane and authenticate to GCP through the Workload Identity binding added in #1823. Three Flux Kustomizations mirroring the AWS chain, since the ordering constraints are identical — controller must run before a Provider installs, and the Provider must be healthy (wait: true) before a ClusterProviderConfig referencing its CRDs will apply. Facts verified by unpacking the v2.6.0 packages rather than inferred by analogy with AWS, because two of them differ: - ProjectIAMMember ships in provider-gcp-cloudplatform as projectiammembers.cloudplatform.gcp.m.upbound.io. That is the only Kind GCPWorkloadIdentity renders, so it is the only provider installed and the only entry in the activation policy. - GCP's ProviderConfig has no PodIdentity credential source at all. The equivalent is InjectedIdentity — ambient ADC, which under GKE Workload Identity resolves to the KSA's federated identity. - ClusterProviderConfig requires an explicit projectID. AWS derives its account from the caller's credentials; GCP does not. The DeploymentRuntimeConfig names its ServiceAccount provider-gcp, which is the subject hard-coded in the OpenTofu binding. Nothing validates that the two agree, so a rename fails silently — documented at both ends. Functions move to a shared functions/ directory referenced by both clouds. They are cluster-scoped singletons with cloud-agnostic pins, and two copies would let the versions drift, producing compositions that behave differently per cloud. Verified the AWS render is byte-identical before and after the move. Deliberately NOT included: a Configuration package for GCP. The XRDs and Compositions belong in Smana/crossplane-configuration and need a new crossplane-configuration-gcp package released there first; pointing at an unpublished tag would leave the Kustomization failing every reconcile. That cutover is two PRs with prune disabled on the first. Also corrects the crossplane README, whose "validating a composition" section still described rendering compositions that left this repo in #1774. Evidence: validate-manifests.sh exit 0, Valid: 1214, Invalid: 0, Skipped: 0; validate-links.sh and validate-doc-claims.sh exit 0; every ${var} referenced confirmed present in the ConfigMap gke/configure creates. * fix(gcp): app-wizard functions path, and address review findings The bug: moving functions.yaml into its own directory broke apps/platform/app-wizard/wizard.yaml, which carries the path as a live config value (functionsPath, resolved under REPO_ROOT by the initContainer clone). After merge the wizard's crossplane render would have shelled out to a file that no longer exists and /api/render-preview would have degraded to its error path. app-wizard is deployed. Nothing in CI catches it — validate-links.sh only walks Markdown, and .doc-claims.yaml has no crossplane entries. This is the same silent path rot a doc move caused before. I checked Markdown links and kustomize references after the git mv, and not YAML config values. A comment that claimed a verification nothing performs: crossplane-configuration said "a healthy ClusterProviderConfig proves the Workload Identity binding authenticates". It proves nothing. Crossplane never contacts GCP to validate credentials, and this Kustomization sets neither wait: true nor a healthCheck, so Flux reports Ready as soon as the object is accepted. Credentials are first exercised when a managed resource reconciles, and this PR ships no claim that renders one. Rewritten to say the binding is unverified until the first ProjectIAMMember. CI was rendering GCP manifests with a literal ${project_id}: render-bundle.py's FIXTURE_VARS gained none of the five new GCP variables, and unknown names pass through verbatim. They still cleared gate 1 because every target field is a free-form string, so the GCP substitution path was validated without ever being exercised. Added all five; project_number is unquoted so the fixture keeps the int shape the composition has to handle. The bundle now renders real values. A comment that invited breaking the bootstrap: crossplane-controller's `dependsOn: crds` was described as symmetry with AWS. It is load-bearing — crds depends on flux-sources, which defines the crossplane HelmRepository, and on namespaces, which creates crossplane-system. Dropping it fails at a point that looks unrelated to that file. Stale after the move: configuration-packages.yaml said functions.yaml was "in this directory"; app-wizard/app.yaml pointed a version-mirror comment at the old path. Comment reduction, ~40 lines with no information lost: the two-PR cutover rationale was stated in full three times (now once in the README, pointers elsewhere); deploymentruntimeconfig-gcp restated 18 lines of iam.tf, and iam.tf still framed the ServiceAccount name as a future obligation rather than one this PR discharges — both now point at each other; environmentconfig instructed KCL authored in another repo, reworded to record what that module does; the passthrough overlay comment restated its own resources: line. Not done: a durable guard for the path class that broke here. .doc-claims.yaml asserts prose matches a config value and cannot assert a path exists, so an entry would give false confidence. Worth a separate check. Evidence: validate-manifests.sh exit 0, Valid: 1214, Invalid: 0, Skipped: 0; validate-links.sh and validate-doc-claims.sh exit 0; tofu fmt clean; bundle confirmed to carry substituted GCP values.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First step of design slice 5 (
GCPWorkloadIdentity): give Crossplane a GCPidentity, scoped rather than broad.
Single file. The rest of slice 5 — the Crossplane Flux tree here, and the XRD in
Smana/crossplane-configuration— follows separately; see What this does notdo.
What changed
opentofu/gcp/gke/init/iam.tfwas a comment-only file. #1818 removed theroles/editorgrant it used to carry, because Crossplane was not deployed onGCP and the binding had no consumer — it was a project-wide editor grant to
a ServiceAccount nothing created, reachable because
namespaces/basecreatescrossplane-systemon GCP too.Slice 5 gives it a consumer, so it comes back — as one binding:
Why the condition is the substance
projectIamAdminalone would be a large improvement oneditorand still aprivilege-escalation path:
setIamPolicycan grant any role to any principal,including granting Crossplane itself
roles/owner.It is also the honest GCP analogue of the AWS side's
xplane-*scoping. AWSrestricts Crossplane by resource name; GCP cannot for project IAM, because
the resource is the project — so it restricts by grantable role instead.
Worth stating plainly, because the earlier conclusion recorded in this repo was
that GCP had no equivalent. It does; it is on a different axis.
Adding a role to the allowlist is therefore deliberate. A workload needing
something outside it fails with a permission error naming the role, rather than
Crossplane quietly having had it all along.
Three things measured, not read
Each would have been a silent failure, and none is catchable by
tofu validate.1. GCP IAM conditions use a restricted CEL dialect. The first attempt allowed
predefined roles or
xplane--prefixed custom ones via.all(...). The APIrejected it at apply:
hasOnly()is the supported form and matches exact strings only.2. That forced a tighter design, which is the better one. Exact names mean a
dynamically-named custom role cannot be allowlisted, so
roles/iam.roleAdminis not granted at all —
GCPWorkloadIdentity's optionalcustomRole.permissionsis deferred rather than half-enabled. Slice 5's actualneed (criterion 21: external-dns records, cert-manager DNS-01) is served
entirely by
roles/dns.admin. The file records what re-enabling it wouldrequire, and that widening the condition is not it.
3. The subject is the PROVIDER's ServiceAccount, not Crossplane's. The first
commit here named
sa/crossplane. That could never have matched — Crossplanecore never talks to GCP; the provider pod makes the cloud API calls. AWS has
bound
crossplane-system/provider-awsall along(
opentofu/aws/eks/init/iam.tf:56-57). Found by checking what AWS binds whilewriting the provider manifests, not from any error — a wrong subject is
accepted by the API and simply never matches, surfacing later as permission
denials that point at the workload rather than the binding.
The corrected name must equal the
serviceAccountTemplatein aDeploymentRuntimeConfigthat does not exist yet, and nothing checks that theyagree — so it is tied to its source in a comment.
Also settled while scoping (no code here, recorded for the next step)
cloudplatform.gcp.m.upbound.iowhile noting the docs say.upbound.io. Thepackage ships both, all 32 CRDs in each flavour; the
ManagedResourceActivationPolicydecides..m.upbound.iois correct — itmatches the AWS policy and Crossplane v2's namespaced MRs.
provider-gcp-cloudplatform:v2.6.0exists, matching every AWS provider.ClusterProviderConfigisgcp.m.upbound.io,credentials.source: InjectedIdentity, plus a GCP-onlyprojectIDfield AWS has no equivalent for.What this does not do
provider,
DeploymentRuntimeConfignamingprovider-gcp, activation policy,ClusterProviderConfig.GCPWorkloadIdentityXRD. It is authored inSmana/crossplane-configurationunderapis/gcpworkloadidentity/and needs anew
crossplane-configuration-gcppackage; only-aws:v0.1.0existstoday. The design warns that migrating a live cluster onto a package is a
two-PR operation with
prune: disabledfirst, or Flux prunes the XRDs anddestroys every claim.
Verification
Applied against the live project, then torn down:
gcloud projects get-iam-policyshowed exactly one crossplane binding, withthe condition attached and the member in the precise
principal://.../ns/crossplane-system/sa/provider-gcpform design criterion19 requires.
sa/crossplanebinding was destroyed, not left alongside.OpenTofu-managed and went with the stack.
tofu validateandtofu fmtclean.