Skip to content

feat(gcp): scoped Crossplane IAM identity for slice 5 - #1823

Merged
Smana merged 5 commits into
mainfrom
worktree-gcp-workload-identity
Aug 24, 2026
Merged

feat(gcp): scoped Crossplane IAM identity for slice 5#1823
Smana merged 5 commits into
mainfrom
worktree-gcp-workload-identity

Conversation

@Smana

@Smana Smana commented Aug 24, 2026

Copy link
Copy Markdown
Owner

First step of design slice 5 (GCPWorkloadIdentity): give Crossplane a GCP
identity, scoped rather than broad.

Single file. The rest of slice 5 — the Crossplane Flux tree here, and the XRD in
Smana/crossplane-configuration — follows separately; see What this does not
do
.

What changed

opentofu/gcp/gke/init/iam.tf was a comment-only file. #1818 removed the
roles/editor grant it used to carry, because Crossplane was not deployed on
GCP and the binding had no consumer — it was a project-wide editor grant to
a ServiceAccount nothing created, reachable because namespaces/base creates
crossplane-system on GCP too.

Slice 5 gives it a consumer, so it comes back — as one binding:

roles/resourcemanager.projectIamAdmin
  member:    ns/crossplane-system/sa/provider-gcp
  condition: modifiedGrantsByRole.hasOnly(['roles/dns.admin'])

Why the condition is the substance

projectIamAdmin alone would be a large improvement on editor and still a
privilege-escalation path
: setIamPolicy can grant any role to any principal,
including granting Crossplane itself roles/owner.

It is also the honest GCP analogue of the AWS side's xplane-* scoping. AWS
restricts Crossplane by resource name; GCP cannot for project IAM, because
the resource is the project — so it restricts by grantable role instead.
Worth stating plainly, because the earlier conclusion recorded in this repo was
that GCP had no equivalent. It does; it is on a different axis.

Adding a role to the allowlist is therefore deliberate. A workload needing
something outside it fails with a permission error naming the role, rather than
Crossplane quietly having had it all along.

Three things measured, not read

Each would have been a silent failure, and none is catchable by tofu validate.

1. GCP IAM conditions use a restricted CEL dialect. The first attempt allowed
predefined roles or xplane--prefixed custom ones via .all(...). The API
rejected it at apply:

Condition expression compilation failed:
undeclared reference to '@not_strictly_false'

hasOnly() is the supported form and matches exact strings only.

2. That forced a tighter design, which is the better one. Exact names mean a
dynamically-named custom role cannot be allowlisted, so roles/iam.roleAdmin
is not granted at all
GCPWorkloadIdentity's optional
customRole.permissions is deferred rather than half-enabled. Slice 5's actual
need (criterion 21: external-dns records, cert-manager DNS-01) is served
entirely by roles/dns.admin. The file records what re-enabling it would
require, and that widening the condition is not it.

3. The subject is the PROVIDER's ServiceAccount, not Crossplane's. The first
commit here named sa/crossplane. That could never have matched — Crossplane
core never talks to GCP; the provider pod makes the cloud API calls. AWS has
bound crossplane-system/provider-aws all along
(opentofu/aws/eks/init/iam.tf:56-57). Found by checking what AWS binds while
writing the provider manifests, not from any error — a wrong subject is
accepted by the API and simply never matches, surfacing later as permission
denials that point at the workload rather than the binding.

The corrected name must equal the serviceAccountTemplate in a
DeploymentRuntimeConfig that does not exist yet, and nothing checks that they
agree — so it is tied to its source in a comment.

Also settled while scoping (no code here, recorded for the next step)

  • The API group question in the design was a false dichotomy. It expected
    cloudplatform.gcp.m.upbound.io while noting the docs say .upbound.io. The
    package ships both, all 32 CRDs in each flavour; the
    ManagedResourceActivationPolicy decides. .m.upbound.io is correct — it
    matches the AWS policy and Crossplane v2's namespaced MRs.
  • provider-gcp-cloudplatform:v2.6.0 exists, matching every AWS provider.
  • ClusterProviderConfig is gcp.m.upbound.io, credentials.source: InjectedIdentity, plus a GCP-only projectID field AWS has no equivalent for.

What this does not do

  • No Crossplane on GCP yet: controller (reusable as-is — it is cloud-agnostic),
    provider, DeploymentRuntimeConfig naming provider-gcp, activation policy,
    ClusterProviderConfig.
  • No GCPWorkloadIdentity XRD. It is authored in
    Smana/crossplane-configuration under apis/gcpworkloadidentity/ and needs a
    new crossplane-configuration-gcp package; only -aws:v0.1.0 exists
    today. The design warns that migrating a live cluster onto a package is a
    two-PR operation with prune: disabled first, or Flux prunes the XRDs and
    destroys every claim.

Verification

Applied against the live project, then torn down:

  • gcloud projects get-iam-policy showed exactly one crossplane binding, with
    the condition attached and the member in the precise
    principal://.../ns/crossplane-system/sa/provider-gcp form design criterion
    19
    requires.
  • The stale sa/crossplane binding was destroyed, not left alongside.
  • After teardown: 0 billable resources, and no IAM leftovers — the binding is
    OpenTofu-managed and went with the stack.
  • tofu validate and tofu fmt clean.

Smana added 2 commits August 24, 2026 13:45
Restores the binding removed in #1818 -- which had no consumer then and does
now -- with least privilege rather than breadth.

BEFORE: roles/editor, project-wide. Thousands of permissions across every
service, granted to a principal nothing created.

AFTER: one binding, roles/resourcemanager.projectIamAdmin, conditioned so it may
grant only an allowlist:

  api.getAttribute('iam.googleapis.com/modifiedGrantsByRole', [])
    .hasOnly(['roles/dns.admin'])

WHY THE CONDITION IS THE POINT

projectIamAdmin alone would be a large improvement on editor and STILL a
privilege-escalation path: setIamPolicy can grant any role to any principal,
including granting Crossplane itself roles/owner. The condition closes that.

It is also the honest GCP analogue of the AWS side's `xplane-*` scoping. AWS
restricts Crossplane by resource NAME; GCP cannot for project IAM, because the
resource IS the project -- so it restricts by grantable ROLE instead. Same
intent, different axis, and worth stating because the earlier conclusion in this
repo was that GCP simply had no equivalent.

TWO THINGS MEASURED, NOT READ

1. GCP IAM conditions run a RESTRICTED CEL dialect. The first attempt used
   `.all(r, r in [...] || r.startsWith(...))` to allow both predefined roles and
   xplane-prefixed custom ones. GCP rejected it at apply:

     Condition expression compilation failed:
     undeclared reference to '@not_strictly_false'

   The `.all()` macro is unavailable; `hasOnly()` is the supported form and
   matches exact strings only. tofu validate passes on the broken version --
   only the API knows.

2. That forced a tighter design, which is the better one. Exact names mean
   dynamically-named custom roles cannot be allowlisted, so roles/iam.roleAdmin
   is NOT granted at all: GCPWorkloadIdentity's optional customRole.permissions
   is deferred rather than half-enabled. Slice 5's actual need -- criterion 21,
   external-dns records and cert-manager's DNS-01 challenge -- is served
   entirely by roles/dns.admin. The file records what re-enabling customRole
   would require, and that widening the condition is not it.

Both traps from the removed version are carried forward: the NUMBER/ID split in
the principal string (reversed, the API accepts it and it never matches), and
the missing graph edge to module.gke that fails only a FRESH apply.

Verified on the live project: `gcloud projects get-iam-policy` shows exactly one
crossplane binding, with the condition attached and the member in the precise
principal://.../ns/crossplane-system/sa/crossplane form design criterion 19
requires. No roles/iam.roleAdmin binding exists.
The binding committed in 33a02f6 could never have matched. It named
`ns/crossplane-system/sa/crossplane`, but Crossplane core never talks to GCP --
the PROVIDER pod makes the cloud API calls, under its own ServiceAccount.

The AWS side has said so all along: opentofu/aws/eks/init/iam.tf:56-57 binds its
Pod Identity to crossplane-system/provider-aws, not to crossplane. I found this
by checking what AWS actually binds while writing the GCP provider manifests,
not from any error -- and there would not have been a useful one.

This is TRAP 1 from the same file, in its other form. A wrong subject is
ACCEPTED by the API and simply never matches: no validation failure, no
Crossplane error at install, just permission denials at first use that point at
the workload rather than at the binding.

The name is now tied to its source in a comment: it must equal the
serviceAccountTemplate in the DeploymentRuntimeConfig that slice 5's provider
tree will carry. The two are set in different repositories' worth of context and
nothing checks that they agree.

Verified on the live project: the stale sa/crossplane binding is destroyed and
exactly one remains --
  roles/resourcemanager.projectIamAdmin
  ns/crossplane-system/sa/provider-gcp
  condition: modifiedGrantsByRole.hasOnly(['roles/dns.admin'])
which is the precise principal:// form design criterion 19 requires.
@github-actions

github-actions Bot commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

🔍 Rendered manifest diff — this PR vs main (desired state)

No changes to the rendered desired state. ✅

Smana added 3 commits August 24, 2026 14:17
Records a decision that workstream 11 would otherwise have had to make under
pressure, and separates two workstreams that are easy to conflate.

THE SPLIT, which was the first thing to get straight:

  10  PUBLIC certs  -- cert-manager clouddns DNS-01 against Let's Encrypt.
                       Depends on slice 5.
  11  PRIVATE certs -- OpenBao's own PKI, the GCP counterpart to what
                       bao.priv.aws.ogenki.io serves. Depends only on
                       workstream 1, so it is NOT blocked by slice 5.

Only 11 needs OpenBao running. Both rows now cross-reference the new section
instead of leaving the reader to infer which is which.

DECISION: two OpenBaos, two roots.

Rejected -- a shared OpenBao reached over the tailnet: zero new infrastructure
and one trust anchor, but it makes GCP certificate issuance hard-depend on AWS
and on the tailnet, in a platform whose stated point is that each cloud stands
alone. The design already flags that same coupling as undesirable for the Flux
GitHub App secret.

Rejected -- two OpenBaos sharing one root: one trust anchor AND independent
operation, which looks like the best of both. It is not, operationally. It needs
either the root PRIVATE KEY copied into GCP Secret Manager, doubling exposure of
the most sensitive material the platform holds, or GCP's intermediate
cross-signed at bootstrap -- a manual ceremony on EVERY REBUILD of a platform
whose lifecycle is build-validate-destroy.

The deciding evidence is from this session rather than from theory. The private
domain rename forced a new OpenBao server certificate, and re-issuing it under
the existing chain turned out to be impossible: the intermediate that had signed
it had no private key stored anywhere, because OpenBao issued it and the key
never left OpenBao. A fresh CA was the only way forward.

That is precisely the failure option C institutionalises, and it fires at
rebuild time -- the worst moment. Two independent roots cost one extra trust
anchor and delete the whole class.

Consequences carried into workstream 11 rather than left implicit: tailnet
clients must trust both roots; GCP needs its own Secret Manager entries for the
root token and cert-manager AppRole, mirroring AWS Secrets Manager and following
the pattern flux-github-app already set; Cloud KMS auto-unseal is what makes an
unattended rebuild possible; and the per-cloud private domains from ADR-0017
make the split clean, since no name resolves to either CA ambiguously.

Nothing here reaches application manifests. Workloads request a cert-manager
Certificate, which is already cloud-neutral -- the issuer differs per cloud, the
developer-facing API does not, which is ADR-0007's split by audience.

Also added to the resume plan, since workstream 11 can start independently and
whoever picks it up will look there first.

Verified: ./scripts/validate-links.sh -> all relative links resolve.
Seven findings applied. The first was a wording error that turned out to be
covering a hole in the plan.

THE CONTRADICTION, AND WHAT WAS BEHIND IT

The new PKI section said workstream 10 issues PUBLIC certificates "for names
under priv.gcp.ogenki.io" -- while the same section, and ADR-0017, say
priv.gcp.ogenki.io is the PRIVATE zone and public stays cloud.ogenki.io. A
section written to stop workstreams 10 and 11 being conflated opened by
conflating them.

Correcting the domain exposed the actual problem: WORKSTREAM 10's DNS-01 HAS
NOTHING TO SOLVE AGAINST. opentofu/gcp/network/dns.tf creates a PRIVATE Cloud
DNS zone and nothing else, and Let's Encrypt must resolve the _acme-challenge
TXT record publicly. cloud.ogenki.io is a Route53 zone this repository does not
even manage -- it appears only as a data lookup. So "cert-manager clouddns
DNS-01" was never a complete plan, and asserting the wrong domain hid that.

Three ways out are now recorded, none chosen: solve DNS-01 against Route53 from
GCP (works today, reintroduces the cross-cloud dependency option A was rejected
for); delegate a public subdomain to a new public Cloud DNS zone; or serve no
public certificates from GCP and keep public ingress on AWS, which is coherent
while GCP has no public endpoints. Worth settling before workstream 10 starts
rather than during it. Workstream 11 is unaffected.

THE DEAD REFERENCE

iam.tf pointed at "the DeploymentRuntimeConfig's serviceAccountTemplate in
infrastructure/gcp-mycluster-0/crossplane/providers/" in the present tense. That
directory does not exist, and `provider-gcp` appears nowhere else in the repo --
a reader greps, finds nothing, and cannot tell whether the binding or the path
is wrong. Restated as what it is: an OBLIGATION on slice 5, with the warning
that nothing checks the two agree.

DUPLICATION, the pattern that keeps recurring here

Four restatements removed: a verbatim re-telling of TRAP 1 twelve lines below
the original; a forward-pointer to a block in the same file; a second copy of
the dns.admin rationale; and a re-summary of the 2026-08-23 OpenBao correction
that already sits earlier in the same document. The resume plan also re-told the
design's whole rationale after naming it as the source -- now reduced to the
sequencing facts that are the plan's actual job, with a link for the reasoning.

The measured-behaviour comments are untouched: the CEL .all() rejection, the
NUMBER/ID split, the provider-SA trap, additive-vs-authoritative. Those are the
file's value; the findings were about copies of them.

THE REVIEWER'S CLOSING CHALLENGE, ACCEPTED

This binding again lands ahead of its consumer -- exactly the situation #1818
removed the old one over. The header no longer glosses that. It states the
caveat, why it is acceptable here where roles/editor was not (blast radius is
one grantable role, not owner; and the provider cannot authenticate without it,
so it must precede the deployment), and instructs removing it again if slice 5
stalls.

Verified: tofu validate and tofu fmt clean; ./scripts/validate-links.sh -> all
relative links resolve.
Security review of the slice 5 Crossplane binding found that roles/dns.admin
carries far more than external-dns and cert-manager need:

- dns.managedZones.delete contradicts the platform constitution's "no deletion
  permissions for stateful services (S3, IAM, Route53)". The AWS side honours
  that rule; this did not.
- The dns.responsePolicies.* / dns.policies.* family lets a compromised
  provider-gcp bind a response policy to the cluster VPC, overriding
  metadata.google.internal or *.googleapis.com to redirect in-cluster traffic
  and harvest credentials — invisible to external-dns.

Replace it with a pre-created xplane_dns_editor custom role holding only
record-set management, transactional changes, and read-only zone lookup.
Pre-creating it in OpenTofu also gives it a deterministic name, which is what
makes it allowlistable at all: the IAM condition matches exact names via
hasOnly, so a composition-rendered role never could be.

Also grant a read-only xplane_role_reader. projectIamAdmin's 9 permissions do
not include iam.roles.get, and referencing a custom role in setIamPolicy can
require reading it — a constraint the predefined-role draft would never have
hit. Whether GCP enforces it here is unverified (needs a live provider pod),
so it is granted pre-emptively: the downside is asymmetric and reading role
definitions confers nothing.

Record three gaps the condition cannot close rather than leaving them assumed
safe: it gates 1 of the role's 9 permissions (modifiedGrantsByRole is undefined
for non-setIamPolicy verbs, making hasOnly vacuously true — latent until the org
adopts Principal Access Boundaries); it constrains the role and never the
member; and it is scoped to the project-wide workload identity pool, so every
future cluster in the project inherits it. The last was documented in #1818 and
lost in the rewrite.

Confirmed while reviewing: modifiedGrantsByRole covers revocations as well as
grants, so the condition also stops Crossplane removing bindings it did not
create, including break-glass human access.

Update the design's example claim, which asked for roles/dns.admin and would
now be refused by the condition, and mark customRole.permissions unusable.
@Smana
Smana merged commit 87211b0 into main Aug 24, 2026
8 checks passed
@Smana
Smana deleted the worktree-gcp-workload-identity branch August 24, 2026 12:37
Smana added a commit that referenced this pull request Aug 24, 2026
* feat(gcp): Crossplane controller, provider and config tree

Slice 5's cluster-side plumbing: the GKE cluster can now run Crossplane and
authenticate to GCP through the Workload Identity binding added in #1823.

Three Flux Kustomizations mirroring the AWS chain, since the ordering
constraints are identical — controller must run before a Provider installs, and
the Provider must be healthy (wait: true) before a ClusterProviderConfig
referencing its CRDs will apply.

Facts verified by unpacking the v2.6.0 packages rather than inferred by analogy
with AWS, because two of them differ:

- ProjectIAMMember ships in provider-gcp-cloudplatform as
  projectiammembers.cloudplatform.gcp.m.upbound.io. That is the only Kind
  GCPWorkloadIdentity renders, so it is the only provider installed and the only
  entry in the activation policy.
- GCP's ProviderConfig has no PodIdentity credential source at all. The
  equivalent is InjectedIdentity — ambient ADC, which under GKE Workload
  Identity resolves to the KSA's federated identity.
- ClusterProviderConfig requires an explicit projectID. AWS derives its account
  from the caller's credentials; GCP does not.

The DeploymentRuntimeConfig names its ServiceAccount provider-gcp, which is the
subject hard-coded in the OpenTofu binding. Nothing validates that the two
agree, so a rename fails silently — documented at both ends.

Functions move to a shared functions/ directory referenced by both clouds. They
are cluster-scoped singletons with cloud-agnostic pins, and two copies would let
the versions drift, producing compositions that behave differently per cloud.
Verified the AWS render is byte-identical before and after the move.

Deliberately NOT included: a Configuration package for GCP. The XRDs and
Compositions belong in Smana/crossplane-configuration and need a new
crossplane-configuration-gcp package released there first; pointing at an
unpublished tag would leave the Kustomization failing every reconcile. That
cutover is two PRs with prune disabled on the first.

Also corrects the crossplane README, whose "validating a composition" section
still described rendering compositions that left this repo in #1774.

Evidence: validate-manifests.sh exit 0, Valid: 1214, Invalid: 0, Skipped: 0;
validate-links.sh and validate-doc-claims.sh exit 0; every ${var} referenced
confirmed present in the ConfigMap gke/configure creates.

* fix(gcp): app-wizard functions path, and address review findings

The bug: moving functions.yaml into its own directory broke
apps/platform/app-wizard/wizard.yaml, which carries the path as a live config
value (functionsPath, resolved under REPO_ROOT by the initContainer clone).
After merge the wizard's crossplane render would have shelled out to a file
that no longer exists and /api/render-preview would have degraded to its error
path. app-wizard is deployed. Nothing in CI catches it — validate-links.sh only
walks Markdown, and .doc-claims.yaml has no crossplane entries.

This is the same silent path rot a doc move caused before. I checked Markdown
links and kustomize references after the git mv, and not YAML config values.

A comment that claimed a verification nothing performs: crossplane-configuration
said "a healthy ClusterProviderConfig proves the Workload Identity binding
authenticates". It proves nothing. Crossplane never contacts GCP to validate
credentials, and this Kustomization sets neither wait: true nor a healthCheck,
so Flux reports Ready as soon as the object is accepted. Credentials are first
exercised when a managed resource reconciles, and this PR ships no claim that
renders one. Rewritten to say the binding is unverified until the first
ProjectIAMMember.

CI was rendering GCP manifests with a literal ${project_id}: render-bundle.py's
FIXTURE_VARS gained none of the five new GCP variables, and unknown names pass
through verbatim. They still cleared gate 1 because every target field is a
free-form string, so the GCP substitution path was validated without ever being
exercised. Added all five; project_number is unquoted so the fixture keeps the
int shape the composition has to handle. The bundle now renders real values.

A comment that invited breaking the bootstrap: crossplane-controller's
`dependsOn: crds` was described as symmetry with AWS. It is load-bearing —
crds depends on flux-sources, which defines the crossplane HelmRepository, and
on namespaces, which creates crossplane-system. Dropping it fails at a point
that looks unrelated to that file.

Stale after the move: configuration-packages.yaml said functions.yaml was "in
this directory"; app-wizard/app.yaml pointed a version-mirror comment at the old
path.

Comment reduction, ~40 lines with no information lost: the two-PR cutover
rationale was stated in full three times (now once in the README, pointers
elsewhere); deploymentruntimeconfig-gcp restated 18 lines of iam.tf, and iam.tf
still framed the ServiceAccount name as a future obligation rather than one this
PR discharges — both now point at each other; environmentconfig instructed KCL
authored in another repo, reworded to record what that module does; the
passthrough overlay comment restated its own resources: line.

Not done: a durable guard for the path class that broke here. .doc-claims.yaml
asserts prose matches a config value and cannot assert a path exists, so an
entry would give false confidence. Worth a separate check.

Evidence: validate-manifests.sh exit 0, Valid: 1214, Invalid: 0, Skipped: 0;
validate-links.sh and validate-doc-claims.sh exit 0; tofu fmt clean; bundle
confirmed to carry substituted GCP values.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant