refactor(gcp): per-cloud OpenTofu state — GCP in GCS, AWS in S3 - #1831
Merged
Conversation
GCP state lived in the AWS S3 bucket alongside every other stack. The reason recorded at the time was sound but argued against the wrong thing: the GCS bucket it replaced sat INSIDE ogenki-435905, the very project whose resources it tracked, so deleting that project would have taken the state describing it along too. That is an argument against a state bucket in the WORKLOAD project, not against GCS. Moving to S3 fixed the self-reference by introducing a different coupling, and the backend comment named it as an accepted cost: "running the GCP stacks now requires AWS credentials as well as GCP ones, and an S3 outage blocks GCP applies." Three things make that worth reversing: 1. GCP stacks now need GCP credentials only. The same cross-cloud coupling was rejected on principle elsewhere in this platform -- the GCP OpenBao design turned down sharing AWS's OpenBao precisely because it would make GCP certificate issuance depend on AWS. This was that coupling under all of it. 2. Teardown survives an AWS outage. A suspended AWS account previously blocked GCP `destroy` as well as `apply`, and teardown is the operation most needed when something is already wrong. 3. openbao/management's state holds cert-manager's live AppRole secret_id -- its own secrets.tf says so. Under a shared bucket an AWS-side compromise handed over a working GCP credential. Project `ogenki-tfstate` holds the bucket and nothing else, which closes the original self-reference loop properly rather than relocating it. `opentofu/shared/*` deliberately stays in S3: the tailnet belongs to neither cloud, and that is the one case the single-bucket rationale gets exactly right. Safe because no GCP stack reads AWS state and no AWS stack reads GCP state -- every terraform_remote_state in the GCP tree is GCP -> GCP, so splitting severed no data flow. Verified before touching anything. Done NOW rather than later on purpose. Every GCP state file contained `resources: 0` (network serial=35, gke/init 29, gke/configure 17, openbao/cluster 17, openbao/management 3) because the platform had just been torn down after verifying workstream 11. State migration is normally the risky half of this change; in that window there was nothing to lose track of, so the new backends were initialised clean rather than copied. That window closes as soon as GCP runs anything again. The cost, stated rather than buried: a SECOND hand-created bootstrap prerequisite, neither IaC-managed. Documented in network/backend.tf with the commands, the way the Cloud KMS key ring already is. ADR-0018 records the decision and supersedes the single-bucket rationale. Evidence: - tofu init against GCS -> all 5 stacks OK, backend=gcs confirmed from .terraform/terraform.tfstate; 5 default.tfstate objects present in gs://ogenki-cloud-native-ref-tfstate - tofu validate -> Success on all 5 stacks - trivy config opentofu/gcp -> exit 0 - ./scripts/validate-links.sh -> exit 0 - ./scripts/validate-doc-claims.sh -> exit 0, 4 claims / 8 page checks
Contributor
🔍 Rendered manifest diff — this PR vs
|
Smana
added a commit
that referenced
this pull request
Aug 25, 2026
Task 6 ran end to end against a real cluster and everything was torn down. 7 of 8 criteria PASS, 1 PARTIAL. The two fixes this branch's reviews produced were both exercised live: - healthCheckExprs on the Gateways. `infrastructure-gapi` reached Ready after both Gateways were genuinely `Programmed=True`, not on object creation -- which is what the old `healthChecks` form would have done. - The crossplane-configuration edge. `infrastructure` visibly waited on it rather than racing the Configuration package, so the GCPWorkloadIdentity claim applied against an installed CRD. Headline result: external-dns on the google provider created both the A record and its TXT registry (`owner=gcp-0`, `resource=httproute/apps/probe`) and PRUNED both within ~30s of the route being deleted. That is the GCPWorkloadIdentity composition's first consumer working -- claim Ready in kube-system, identity by subject, no key material anywhere. Criterion 7 is PARTIAL and recorded as such rather than claimed. The admin Gateway accepts a kube-system route and serves it over the tailnet with a valid chain, but proving a NON-admin device is denied needs a tailnet device outside group:admin, and the only device available belongs to the tailnet owner. Asserting the denial from configuration would not be evidence. Two defects the deploy found, neither visible to static validation: - opentofu/gcp/openbao/management/variables.tfvars was never committed, so that stack cannot deploy from a clean checkout. Fixed in 9dc52b2; it is a defect in merged main (#1830), invisible because the file existed untracked in the worktree where the stack was first verified. - `kubectl wait --for=condition=Accepted httproute/...` times out on a healthy route -- an HTTPRoute's conditions live under status.parents[], not at the top level. Same shape as the Gateway healthChecks finding: a readiness assertion that silently checks nothing. Also records two credentials the plan never listed as prerequisites: AWS credentials (needed by a GCP-only deploy purely because GCP state still lives in S3 -- the coupling #1831 removes, demonstrated rather than argued) and TF_VAR_tailscale_api_key. Teardown verified against the GCP API rather than an exit code: 0 instances, 0 clusters, 0 forwarding rules, 0 addresses, 0 disks, the DNS zone gone, and 0 tailnet devices left behind. No pvc-* disk leak this time, unlike the previous GCP teardown.
Smana
added a commit
that referenced
this pull request
Aug 25, 2026
* docs(gcp): design for private ingress and external-dns on GCP Workstream 10 of the GCP support design, scoped down from what the parent design's table implied. Two things shrank it. The tailnet singletons -- ACLs, tagOwners for tag:k8s / tag:admin / tag:k8s-operator, autoApprovers, search domains, advertised routes -- were already extracted into opentofu/shared/tailscale so neither cloud authorises the other's devices. So this workstream changes no ACL at all. And the Gateway API layer is already cloud-neutral: six of the seven manifests in infrastructure/base/gapi render correctly on GCP unchanged, including the wildcard Certificate, which asks the `openbao` ClusterIssuer that workstream 11 verified. What is left is three cluster-side components: the tailscale-operator, the Gateway API layer consumed by file, and external-dns with the google provider. PUBLIC certificates are deliberately excluded. The parent design's open question already observed that DNS-01 has nothing to solve against on GCP -- the Cloud DNS zone is private, cloud.ogenki.io is a Route53 zone this repo does not manage, and Let's Encrypt must resolve the challenge publicly. It stays open because it is not blocking: GCP has no public endpoint to put a certificate on, and workstream 12 owns that. Settling a registrar-level decision on speculation is the failure mode here. external-dns gets its permissions through the GCPWorkloadIdentity composition, which is the opposite call from workstream 11 and deliberately so. There, External Secrets needed exactly two named secrets and the composition's project-scoped ProjectIAMMember would have granted read of every secret in the project, including the intermediate CA's private key -- so it was bound per secret and the composition was left without a consumer. external-dns's access genuinely IS project-shaped: it must discover which zone owns a name, which needs managedZones.list across the project and cannot be expressed per zone. Records two costs rather than burying them: the Gateway hostname change touches AWS, and manual bootstrap steps are now accumulating (Cloud KMS key ring, the OpenTofu state bucket and its project, and now a Tailscale OAuth client) -- three undocumented steps is how a repository stops being reproducible, so the design requires them collected into one procedure. ./scripts/validate-links.sh -> exit 0 * docs(gcp): implementation plan for private ingress on gcp-0 Six tasks. Task 1 parameterises the Gateway Tailscale hostnames (the one change touching AWS-consumed manifests), 2 creates the bootstrap OAuth client plus a single consolidated docs/gcp-bootstrap.md, 3-5 build the three cluster-side components, and 6 verifies live against all eight success criteria and tears everything down. Also fixes two broken links in the design, caught by validate-links.sh rather than by reading: ADR-0005's filename was invented (0005-gke-standard-with-cilium.md; the real one is 0005-gke-standard-self-managed-cilium.md), and ADR-0018 is on the unmerged #1831 branch so it cannot be linked from here yet. The plan carries three things forward that this session paid for: - Ordering edges are dependsOn plus health checks, not retry timers. Workstream 11 shipped a layer whose ExternalSecrets sat unsynced behind an admission webhook with no endpoints, backing off exponentially while Flux reported Ready. - Verification reads logs and re-queries the cloud API, never exit codes. A teardown in this repo has already reported success while destroying nothing. - Task 6 ends with a verified teardown, per the standing rule that no test infrastructure survives. ./scripts/validate-links.sh -> exit 0 * docs(gcp): pre-flight corrections to the private-ingress plan Three defects found by the SDD pre-flight scan before any task was dispatched, two of them in the plan I wrote: - Task 5 patched `global.imageRegistry` and justified it with a claim that the base makes GCP image pulls depend on an AWS registry. Verified against `helm show values external-dns/external-dns --version 1.21.1`: neither `global.imageRegistry` nor a top-level `aws:` key is a chart value at all (`global` supports only `imagePullSecrets`). Both are inert on both clouds. The override is dropped and the comment now says what is true. - Task 6's ACL-split criterion curled `hubble-gcp-0.priv.gcp.ogenki.io`, but Hubble UI is not deployed on gcp-0 -- the criterion would have tested a hostname that never exists. Replaced with a purpose-made admin probe in kube-system, which is in the admin Gateway's allowedRoutes where `apps` is not. The scan table and all three rulings are in the plan's SDD ledger. * refactor(gapi): per-cluster Tailscale hostnames on the private Gateways * docs(gapi): fix stale Tailscale hostname example, pin it as a doc claim * feat(gcp): Tailscale OAuth bootstrap secret, and one bootstrap doc * feat(gcp): tailscale-operator on gcp-0 * feat(gcp): Tailscale Gateways and the private wildcard certificate * feat(gcp): external-dns on the google provider via GCPWorkloadIdentity * fix(gcp): substitute vars in the infrastructure Kustomization external-dns's claim and HelmRelease under infrastructure/gcp-0/ are the first manifests on that path to reference ${var}; without postBuild the literal text reaches the API server and fails the GCPWorkloadIdentity XRD's roles[] pattern. * fix(gcp): gate Gateway readiness on Programmed, not existence healthChecks on a Gateway only asserts the object exists -- Gateway API v1 has no Ready condition and no top-level observedGeneration, so kstatus reports Current the instant the Gateway is created, before the operator has serviced the LoadBalancer, before Cilium has accepted the GatewayClass, before the TLS secret resolves. Add healthCheckExprs gating on the Programmed condition Cilium actually sets, and correct the comment that claimed the literal healthChecks form already proved the whole chain. Same defective block existed in the plan (the durable artifact once .superpowers/ is gitignored away); fixed there too. Final review finding 1 (Important). * fix(gcp): add crossplane-configuration edge to infrastructure Kustomization infrastructure/gcp-0's path now applies a GCPWorkloadIdentity claim (for external-dns) whose CRD and gke-environment EnvironmentConfig both come from the crossplane-configuration Kustomization, three stages down that chain. The edge was missing, so a fresh cluster's `infrastructure` reconciles before the Configuration package lands and fails with `no matches for kind "GCPWorkloadIdentity"` until it self-heals on retry. Add `- name: crossplane-configuration` to dependsOn (minimum fix; not the per-claim-Kustomization restructure the review floated as "better" -- out of scope for this wave). Also rewrite the two comments in the same file that described this path as holding "only a ComputeClass" -- it now also holds a namespaced HelmRelease and a namespaced Crossplane claim, which is exactly why the missing edge went unnoticed. Correct the plan's matching false sentence ("Its claim does need Crossplane, which `infrastructure` already sequences behind") -- that sentence is why nobody added the edge in the first place. Final review findings 2 and 3 (Important, Minor). * fix(security): parameterize tailscale-operator device hostname per cluster The chart default (OPERATOR_HOSTNAME=tailscale-operator, verified via `helm template tailscale/tailscale-operator --version 1.90.6`) is not tailnet-unique. Both aws-0 and gcp-0 now run this shared HelmRelease against one shared tailnet, so gcp-0's operator claims a hostname aws-0's operator already holds and Tailscale silently suffixes one of them -- the same collision the Gateway hostnames were already parameterized against. Fixed in the shared HelmRelease, not a GCP-only patch, so cluster_name substitutes identically on both clouds and the AWS operator device is renamed once, in the same cutover window the Gateway rename already requires. Final review finding 4 (Minor). * fix(docs): correct two comments that overstated what they describe infrastructure/gcp-0/external-dns/kustomization.yaml: the base sets no `provider` key at all (`provider: aws` is the chart default, not something the base writes) and global.imageRegistry is documented three files away, in this directory's own helmrelease.yaml, as inert on chart 1.21.1 -- so this comment contradicted its sibling file. Reduce the parenthetical to what the base actually sets: two domain filters and an inert `aws:` block. security/gcp-0/tailscale-operator/kustomization.yaml: record, rather than silently leave unconverged, that the two ProxyClasses' CRD arrives from crds-tailscale-operator, a child Flux Kustomization the parent `crds` does not wait on -- the same gap clusters/gcp-0/security/security-openbao.yaml already documents for the ExternalSecret CRDs, in the same shape. Final review findings 5 and 6 (Minor). * fix(docs): three deferred-minor accuracy fixes on this branch - private-access.md: the mechanism is postBuild.substituteFrom, not postBuild.substitute. - design doc: the OAuth Secret Manager entry is `tailscale-k8s-operator-oauth` everywhere else (plan, docs/gcp-bootstrap.md, the ExternalSecret, variables.tf) -- drop the stray `-gcp` suffix. - plan Task 5 Step 5 "Expected": the render DOES contain the base's `aws:` block and `zoneType`, deliberately (ruling 2 -- both are inert on chart 1.21.1, so patching them out would be a no-op edit). The plan claimed the opposite, which would send a future reader to "fix" correct code. Final review's three deferred-minor list, marked MUST-FIX. * fix(gcp): track the openbao/management tfvars, missing since #1830 `opentofu/gcp/openbao/management/variables.tfvars` was never committed, so that stack cannot be deployed from a clean checkout: Error: Failed to read variables file Given variables file variables.tfvars does not exist. `*.tfvars` is gitignored repo-wide (.gitignore:55). Every other GCP stack -- network, gke/init, gke/configure, openbao/cluster -- has its variables.tfvars force-added past that rule. This one was missed, and the gap was invisible because the file existed untracked in the worktree where the stack was originally written and verified. Found by deploying workstream 10 from a fresh worktree: network, both GKE stages and openbao/cluster all applied, then management failed instantly. It is a defect in merged main (#1830), not in this branch -- carried here because it blocks this branch's own live verification, and splitting it into its own PR would mean workstream 10 could not be verified at all. Contents are identical to the file the stack was verified against: only project_id, which has no default; everything else stays on the defaults in variables.tf. * docs(gcp): live verification of private ingress on gcp-0 Task 6 ran end to end against a real cluster and everything was torn down. 7 of 8 criteria PASS, 1 PARTIAL. The two fixes this branch's reviews produced were both exercised live: - healthCheckExprs on the Gateways. `infrastructure-gapi` reached Ready after both Gateways were genuinely `Programmed=True`, not on object creation -- which is what the old `healthChecks` form would have done. - The crossplane-configuration edge. `infrastructure` visibly waited on it rather than racing the Configuration package, so the GCPWorkloadIdentity claim applied against an installed CRD. Headline result: external-dns on the google provider created both the A record and its TXT registry (`owner=gcp-0`, `resource=httproute/apps/probe`) and PRUNED both within ~30s of the route being deleted. That is the GCPWorkloadIdentity composition's first consumer working -- claim Ready in kube-system, identity by subject, no key material anywhere. Criterion 7 is PARTIAL and recorded as such rather than claimed. The admin Gateway accepts a kube-system route and serves it over the tailnet with a valid chain, but proving a NON-admin device is denied needs a tailnet device outside group:admin, and the only device available belongs to the tailnet owner. Asserting the denial from configuration would not be evidence. Two defects the deploy found, neither visible to static validation: - opentofu/gcp/openbao/management/variables.tfvars was never committed, so that stack cannot deploy from a clean checkout. Fixed in 9dc52b2; it is a defect in merged main (#1830), invisible because the file existed untracked in the worktree where the stack was first verified. - `kubectl wait --for=condition=Accepted httproute/...` times out on a healthy route -- an HTTPRoute's conditions live under status.parents[], not at the top level. Same shape as the Gateway healthChecks finding: a readiness assertion that silently checks nothing. Also records two credentials the plan never listed as prerequisites: AWS credentials (needed by a GCP-only deploy purely because GCP state still lives in S3 -- the coupling #1831 removes, demonstrated rather than argued) and TF_VAR_tailscale_api_key. Teardown verified against the GCP API rather than an exit code: 0 instances, 0 clusters, 0 forwarding rules, 0 addresses, 0 disks, the DNS zone gone, and 0 tailnet devices left behind. No pvc-* disk leak this time, unlike the previous GCP teardown.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reverses the single-bucket state decision recorded in
opentofu/gcp/network/backend.tfon 2026-08-23. ADR: 0018.Why
The recorded reason for putting GCP state in the AWS S3 bucket was sound but argued against the wrong thing. The GCS bucket it replaced sat inside
ogenki-435905— the very project whose resources it tracked — so deleting that project would have taken the state describing it along too. That is an argument against a state bucket in the workload project, not against GCS.The S3 move fixed the self-reference by introducing a different coupling, and the comment named it as an accepted cost: "running the GCP stacks now requires AWS credentials as well as GCP ones, and an S3 outage blocks GCP applies."
Three things make it worth reversing:
destroyas well asapply, and teardown is the operation most needed when something is already wrong.openbao/managementstate holds a live credential — cert-manager's AppRolesecret_id, as its ownsecrets.tfsays. Under a shared bucket, an AWS-side compromise handed over a working GCP credential.Project
ogenki-tfstateholds the bucket and nothing else, closing the original loop properly instead of relocating it.Scope
opentofu/gcp/**gcsgs://ogenki-cloud-native-ref-tfstate, projectogenki-tfstate, europe-west4opentofu/aws/**s3opentofu/shared/**s3Safe because no GCP stack reads AWS state and no AWS stack reads GCP state. Every
terraform_remote_statein the GCP tree is GCP → GCP (gke/initandopenbao/clusterreadgcp/network;gke/configurereadsgke/init), so splitting severed no data flow. Checked before touching anything.Timing — why now and not later
Every GCP state file contained
resources: 0, because the platform had just been torn down after verifying workstream 11:State migration is normally the risky half of a change like this. In that window there was nothing to lose track of, so the new backends were initialised clean rather than copied. The window closes the moment GCP runs anything again.
The cost, stated rather than buried
A second hand-created bootstrap prerequisite, neither IaC-managed. That was the second recorded reason for one bucket and it does not disappear — it gets documented in
network/backend.tfwith the commands, the way the Cloud KMS key ring already is.The old GCP state objects remain in S3 under
cloud-native-ref/gcp/. They hold no resources, but they are now misleading — anything pointing at those keys reads empty state. Worth deleting once this settles; left in place here rather than destroying state objects in a refactor PR.Evidence
Bucket settings verified:
versioning=True,uniform_bucket_level_access=true,public_access_prevention=enforced.