feat(gcp): private ingress and external-dns on gcp-0 - #1833
Merged
Conversation
Workstream 10 of the GCP support design, scoped down from what the parent design's table implied. Two things shrank it. The tailnet singletons -- ACLs, tagOwners for tag:k8s / tag:admin / tag:k8s-operator, autoApprovers, search domains, advertised routes -- were already extracted into opentofu/shared/tailscale so neither cloud authorises the other's devices. So this workstream changes no ACL at all. And the Gateway API layer is already cloud-neutral: six of the seven manifests in infrastructure/base/gapi render correctly on GCP unchanged, including the wildcard Certificate, which asks the `openbao` ClusterIssuer that workstream 11 verified. What is left is three cluster-side components: the tailscale-operator, the Gateway API layer consumed by file, and external-dns with the google provider. PUBLIC certificates are deliberately excluded. The parent design's open question already observed that DNS-01 has nothing to solve against on GCP -- the Cloud DNS zone is private, cloud.ogenki.io is a Route53 zone this repo does not manage, and Let's Encrypt must resolve the challenge publicly. It stays open because it is not blocking: GCP has no public endpoint to put a certificate on, and workstream 12 owns that. Settling a registrar-level decision on speculation is the failure mode here. external-dns gets its permissions through the GCPWorkloadIdentity composition, which is the opposite call from workstream 11 and deliberately so. There, External Secrets needed exactly two named secrets and the composition's project-scoped ProjectIAMMember would have granted read of every secret in the project, including the intermediate CA's private key -- so it was bound per secret and the composition was left without a consumer. external-dns's access genuinely IS project-shaped: it must discover which zone owns a name, which needs managedZones.list across the project and cannot be expressed per zone. Records two costs rather than burying them: the Gateway hostname change touches AWS, and manual bootstrap steps are now accumulating (Cloud KMS key ring, the OpenTofu state bucket and its project, and now a Tailscale OAuth client) -- three undocumented steps is how a repository stops being reproducible, so the design requires them collected into one procedure. ./scripts/validate-links.sh -> exit 0
Six tasks. Task 1 parameterises the Gateway Tailscale hostnames (the one change touching AWS-consumed manifests), 2 creates the bootstrap OAuth client plus a single consolidated docs/gcp-bootstrap.md, 3-5 build the three cluster-side components, and 6 verifies live against all eight success criteria and tears everything down. Also fixes two broken links in the design, caught by validate-links.sh rather than by reading: ADR-0005's filename was invented (0005-gke-standard-with-cilium.md; the real one is 0005-gke-standard-self-managed-cilium.md), and ADR-0018 is on the unmerged #1831 branch so it cannot be linked from here yet. The plan carries three things forward that this session paid for: - Ordering edges are dependsOn plus health checks, not retry timers. Workstream 11 shipped a layer whose ExternalSecrets sat unsynced behind an admission webhook with no endpoints, backing off exponentially while Flux reported Ready. - Verification reads logs and re-queries the cloud API, never exit codes. A teardown in this repo has already reported success while destroying nothing. - Task 6 ends with a verified teardown, per the standing rule that no test infrastructure survives. ./scripts/validate-links.sh -> exit 0
Three defects found by the SDD pre-flight scan before any task was dispatched, two of them in the plan I wrote: - Task 5 patched `global.imageRegistry` and justified it with a claim that the base makes GCP image pulls depend on an AWS registry. Verified against `helm show values external-dns/external-dns --version 1.21.1`: neither `global.imageRegistry` nor a top-level `aws:` key is a chart value at all (`global` supports only `imagePullSecrets`). Both are inert on both clouds. The override is dropped and the comment now says what is true. - Task 6's ACL-split criterion curled `hubble-gcp-0.priv.gcp.ogenki.io`, but Hubble UI is not deployed on gcp-0 -- the criterion would have tested a hostname that never exists. Replaced with a purpose-made admin probe in kube-system, which is in the admin Gateway's allowedRoutes where `apps` is not. The scan table and all three rulings are in the plan's SDD ledger.
external-dns's claim and HelmRelease under infrastructure/gcp-0/ are the
first manifests on that path to reference ${var}; without postBuild the
literal text reaches the API server and fails the GCPWorkloadIdentity
XRD's roles[] pattern.
healthChecks on a Gateway only asserts the object exists -- Gateway API v1 has no Ready condition and no top-level observedGeneration, so kstatus reports Current the instant the Gateway is created, before the operator has serviced the LoadBalancer, before Cilium has accepted the GatewayClass, before the TLS secret resolves. Add healthCheckExprs gating on the Programmed condition Cilium actually sets, and correct the comment that claimed the literal healthChecks form already proved the whole chain. Same defective block existed in the plan (the durable artifact once .superpowers/ is gitignored away); fixed there too. Final review finding 1 (Important).
…zation
infrastructure/gcp-0's path now applies a GCPWorkloadIdentity claim (for
external-dns) whose CRD and gke-environment EnvironmentConfig both come from
the crossplane-configuration Kustomization, three stages down that chain. The
edge was missing, so a fresh cluster's `infrastructure` reconciles before the
Configuration package lands and fails with `no matches for kind
"GCPWorkloadIdentity"` until it self-heals on retry. Add
`- name: crossplane-configuration` to dependsOn (minimum fix; not the
per-claim-Kustomization restructure the review floated as "better" -- out of
scope for this wave).
Also rewrite the two comments in the same file that described this path as
holding "only a ComputeClass" -- it now also holds a namespaced HelmRelease and
a namespaced Crossplane claim, which is exactly why the missing edge went
unnoticed. Correct the plan's matching false sentence ("Its claim does need
Crossplane, which `infrastructure` already sequences behind") -- that
sentence is why nobody added the edge in the first place.
Final review findings 2 and 3 (Important, Minor).
…uster The chart default (OPERATOR_HOSTNAME=tailscale-operator, verified via `helm template tailscale/tailscale-operator --version 1.90.6`) is not tailnet-unique. Both aws-0 and gcp-0 now run this shared HelmRelease against one shared tailnet, so gcp-0's operator claims a hostname aws-0's operator already holds and Tailscale silently suffixes one of them -- the same collision the Gateway hostnames were already parameterized against. Fixed in the shared HelmRelease, not a GCP-only patch, so cluster_name substitutes identically on both clouds and the AWS operator device is renamed once, in the same cutover window the Gateway rename already requires. Final review finding 4 (Minor).
infrastructure/gcp-0/external-dns/kustomization.yaml: the base sets no `provider` key at all (`provider: aws` is the chart default, not something the base writes) and global.imageRegistry is documented three files away, in this directory's own helmrelease.yaml, as inert on chart 1.21.1 -- so this comment contradicted its sibling file. Reduce the parenthetical to what the base actually sets: two domain filters and an inert `aws:` block. security/gcp-0/tailscale-operator/kustomization.yaml: record, rather than silently leave unconverged, that the two ProxyClasses' CRD arrives from crds-tailscale-operator, a child Flux Kustomization the parent `crds` does not wait on -- the same gap clusters/gcp-0/security/security-openbao.yaml already documents for the ExternalSecret CRDs, in the same shape. Final review findings 5 and 6 (Minor).
- private-access.md: the mechanism is postBuild.substituteFrom, not postBuild.substitute. - design doc: the OAuth Secret Manager entry is `tailscale-k8s-operator-oauth` everywhere else (plan, docs/gcp-bootstrap.md, the ExternalSecret, variables.tf) -- drop the stray `-gcp` suffix. - plan Task 5 Step 5 "Expected": the render DOES contain the base's `aws:` block and `zoneType`, deliberately (ruling 2 -- both are inert on chart 1.21.1, so patching them out would be a no-op edit). The plan claimed the opposite, which would send a future reader to "fix" correct code. Final review's three deferred-minor list, marked MUST-FIX.
Contributor
🔍 Rendered manifest diff — this PR vs
|
`opentofu/gcp/openbao/management/variables.tfvars` was never committed, so that stack cannot be deployed from a clean checkout: Error: Failed to read variables file Given variables file variables.tfvars does not exist. `*.tfvars` is gitignored repo-wide (.gitignore:55). Every other GCP stack -- network, gke/init, gke/configure, openbao/cluster -- has its variables.tfvars force-added past that rule. This one was missed, and the gap was invisible because the file existed untracked in the worktree where the stack was originally written and verified. Found by deploying workstream 10 from a fresh worktree: network, both GKE stages and openbao/cluster all applied, then management failed instantly. It is a defect in merged main (#1830), not in this branch -- carried here because it blocks this branch's own live verification, and splitting it into its own PR would mean workstream 10 could not be verified at all. Contents are identical to the file the stack was verified against: only project_id, which has no default; everything else stays on the defaults in variables.tf.
Task 6 ran end to end against a real cluster and everything was torn down. 7 of 8 criteria PASS, 1 PARTIAL. The two fixes this branch's reviews produced were both exercised live: - healthCheckExprs on the Gateways. `infrastructure-gapi` reached Ready after both Gateways were genuinely `Programmed=True`, not on object creation -- which is what the old `healthChecks` form would have done. - The crossplane-configuration edge. `infrastructure` visibly waited on it rather than racing the Configuration package, so the GCPWorkloadIdentity claim applied against an installed CRD. Headline result: external-dns on the google provider created both the A record and its TXT registry (`owner=gcp-0`, `resource=httproute/apps/probe`) and PRUNED both within ~30s of the route being deleted. That is the GCPWorkloadIdentity composition's first consumer working -- claim Ready in kube-system, identity by subject, no key material anywhere. Criterion 7 is PARTIAL and recorded as such rather than claimed. The admin Gateway accepts a kube-system route and serves it over the tailnet with a valid chain, but proving a NON-admin device is denied needs a tailnet device outside group:admin, and the only device available belongs to the tailnet owner. Asserting the denial from configuration would not be evidence. Two defects the deploy found, neither visible to static validation: - opentofu/gcp/openbao/management/variables.tfvars was never committed, so that stack cannot deploy from a clean checkout. Fixed in 9dc52b2; it is a defect in merged main (#1830), invisible because the file existed untracked in the worktree where the stack was first verified. - `kubectl wait --for=condition=Accepted httproute/...` times out on a healthy route -- an HTTPRoute's conditions live under status.parents[], not at the top level. Same shape as the Gateway healthChecks finding: a readiness assertion that silently checks nothing. Also records two credentials the plan never listed as prerequisites: AWS credentials (needed by a GCP-only deploy purely because GCP state still lives in S3 -- the coupling #1831 removes, demonstrated rather than argued) and TF_VAR_tailscale_api_key. Teardown verified against the GCP API rather than an exit code: 0 instances, 0 clusters, 0 forwarding rules, 0 addresses, 0 disks, the DNS zone gone, and 0 tailnet devices left behind. No pvc-* disk leak this time, unlike the previous GCP teardown.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Workstream 10 of the GCP support design: private ingress for
gcp-0— two Tailscale-backed Gateways serving*.priv.gcp.ogenki.io, TLS from the OpenBao PKI, and DNS records maintained in the private Cloud DNS zone.Design:
2026-08-25-gcp-private-ingress-design.md· Plan:2026-08-25-gcp-private-ingress.mdWhat it builds
Three cluster-side components. The tailnet foundation already existed, so this is smaller than the parent design's table implies — the ACLs,
tagOwners, autoApprovers, search domains and advertised routes all live inopentofu/shared/tailscale/already, and this workstream changes no ACL at all.security/gcp-0/tailscale-operator/— chart + both ProxyClasses referenced by file fromsecurity/base/, with GCP's own OAuth ExternalSecretinfrastructure/gcp-0/gapi/— six cloud-neutral manifests frominfrastructure/base/gapi/, excluding the AWS-shaped public Gatewayinfrastructure/gcp-0/external-dns/—googleprovider, private zone only, identity from aGCPWorkloadIdentityclaimThe wildcard
Certificateneeded no GCP variant: it already asks theopenbaoClusterIssuer that workstream 11 verified end-to-end on this cluster.Using
loadBalancerClass: tailscalerather than a GCP forwarding rule means the private gateways incur no cloud load-balancer charges.What it deliberately does not build
Public certificates and public ingress. The parent design's open question observed that DNS-01 has nothing to solve against on GCP — the Cloud DNS zone is private,
cloud.ogenki.iois a Route53 zone this repo doesn't manage, and Let's Encrypt must resolve the challenge publicly. It stays open because it isn't blocking: GCP has no public endpoint to put a certificate on, and workstream 12 owns that. Settling a registrar-level decision on speculation is the failure mode here.Also no
ProxyGroup— AWS runs two egress proxies; nothing ongcp-0egresses through the tailnet yet.The one change that touches AWS
Both Gateways' Tailscale hostnames become
gateway-{general,admin}-priv-${cluster_name}, and the operator's own device becomestailscale-operator-${cluster_name}. The tailnet is shared and a Tailscale hostname is tailnet-unique — two clusters claiming the same name doesn't error, it silently suffixes one of them.Parameterising rather than suffixing only GCP is deliberate: a scheme where one cloud is the unlabelled default is exactly what ADR-0017 rejects, and is why the clusters were renamed
aws-0/gcp-0in #1832.Cost, stated rather than buried: the AWS operator registers new Tailscale devices on next reconcile. external-dns rewrites the
*.priv.aws.ogenki.iorecords after its sync interval, so there is a brief window where private AWS services are unreachable, and the superseded devices linger in the tailnet and need deleting by hand. Apply this in a window where that is acceptable.Identity — the opposite call from workstream 11, deliberately
external-dns gets its permissions through the
GCPWorkloadIdentitycomposition — its first consumer. Workstream 11 rejected that same composition for External Secrets and bound per-secret instead, becauseProjectIAMMemberis project-scoped and would have granted read of every secret in the project including the intermediate CA's private key.external-dns's access genuinely is project-shaped: it must discover which zone owns a name, which needs
dns.managedZones.listacross the project and cannot be expressed per zone.xplane_dns_editorexcludes zone create and delete. The comments say plainly that the role is project-wide and thatdomainFiltersis a client-side filter, not a security boundary.Defects found and fixed before any deploy
Six reviews ran — one per task plus a whole-branch review. They found two things static validation could not:
postBuild.substituteFrom(Critical).${project_id}would have reached the API server literally, failing theGCPWorkloadIdentityclaim'sroles[]pattern and taking the wholeinfrastructureKustomization down. Found independently by both the implementer and the reviewer.healthCheckson a Gateway asserts existence, not readiness (Important). Gateway API v1 has noReadycondition and no top-levelobservedGeneration, so kstatus reports Current the moment the object exists — and the comment claimed the opposite. Now gated on theProgrammedcondition viahealthCheckExprs, the formclusters/aws-0-llm-platform/security-llm-epi.yamlalready documents.infrastructureKustomization had no edge tocrossplane-configuration(Important) while applying a Crossplane claim, because the plan asserted an edge that did not exist.Two validation blind spots surfaced, both filed as follow-ups rather than fixed here:
./scripts/validate-manifests.shcannot catch a missingpostBuildblock —render-bundle.pysubstitutes its fixtures unconditionally, regardless of whether the real Kustomization declares substitution. The gate passed while the manifest was undeployable.helm templates the GCP external-dns values: renderable HelmReleases are keyed by(namespace, name)and both clouds resolve tokube-system/external-dns, so the bundle renders AWS's values only. Nothing is broken today — the merged GCP values were rendered manually and produce--provider=googlecorrectly.Both are the same shape as the SPEC-007 problem that gate was built to remove.
Gates
Still open
docs/gcp-bootstrap.mdrather than scattered across three files.crossplane-configurationhas no health gate on its Configuration package going Healthy, so the new edge orders but does not prove the XRD exists; the 1m retry is the backstop. A narrower window than before, not the same defect.