Skip to content

fix(gcp): unblock the deploy, and stop --reverse destroy stranding a cluster - #1826

Merged
Smana merged 2 commits into
mainfrom
worktree-gcp-deploy-fixes
Aug 24, 2026
Merged

fix(gcp): unblock the deploy, and stop --reverse destroy stranding a cluster#1826
Smana merged 2 commits into
mainfrom
worktree-gcp-deploy-fixes

Conversation

@Smana

@Smana Smana commented Aug 24, 2026

Copy link
Copy Markdown
Owner

Two workflow defects found by actually running the GCP deploy/verify/teardown cycle today. Both block the documented path; neither is reachable by CI.

1. trivy config blocked the deploy before a single resource was planned

terramate script run deploy runs trivy config --exit-code=1 ahead of apply, and it failed on GCP-0050 ("Cluster does not override the default service account"). Stage 1 never got to plan.

It is a false positive of the same shape as GCP-0057, already documented two lines above it in the same file. Node pools take local.service_account, which the module computes conditionally over create_service_account, so it renders as (known after apply) and trivy assumes the default compute SA.

Verified in the plan rather than read from the module source alone:

module.gke.google_service_account.cluster_service_account[0]                        will be created
module.gke.google_project_iam_member.cluster_service_account_metric_writer[0]
module.gke.google_project_iam_member.cluster_service_account_node_service_account[0]
module.gke.google_project_iam_member.cluster_service_account_resource_metadata_writer[0]

A dedicated account with three narrowly-scoped bindings — not Editor, and not the default compute SA.

I could not establish when this started firing: the stack deployed cleanly on 2026-08-23 and the module version is unchanged at 44.3.0. That is recorded as unknown in the commit rather than dressed up with a theory. The suppression stands on the plan evidence.

2. --reverse destroy stranded a live cluster

terramate script run --reverse destroy from opentofu/gcp/ visits gke/configure before gke/init. That stack's destroy ran a bare tofu destroy under set -euo pipefail, so a failure aborted the run before the billable resource was touched.

Hit for real today, ~63 minutes after cluster creation:

Error: flux-system/gke-gcp-mycluster-0-vars failed to delete kubernetes resource: Unauthorized

The helm and kubectl providers hold a GCP access token acquired at plan time; it had expired. Teardown stopped there, and gcloud container clusters list showed the cluster and both static nodes still RUNNING. Removing them took a second, manual run of gke/init's destroy.

The expired token is the mild version. The usual reasons are worse — a broken cluster, or a private endpoint unreachable because the tailnet is down — which is exactly when a teardown matters most.

gke/init already solved this. Its destroy calls scripts/gke-destroy-stage2.sh in attempt mode (graceful, never fails the caller) and defers state cleanup to a reconcile job that runs only once the cluster is provably gone. That fix landed on gke/init's own destroy and never reached the --reverse path, so the hole reopened through a different door. This points configure's destroy at the same helper, so the two entry points cannot drift apart again.

State is deliberately not cleared here: at this point the cluster may still exist, so state may still be accurate. gke/init's reconcile job drops what is left, and only afterwards.

Evidence

Both fixes were exercised end to end on a real cluster today, then the cluster was destroyed.

Check Result
Deploy with the trivy fix Apply complete! Resources: 21 added, 0 changed, 0 destroyed — cluster, Cilium, Flux Operator, Flux Instance
Teardown via the tolerant path cluster + node pool + IAM destroyed, ending [ok] stage 2 state cleared after reconciling 23 stale instances out of state
terramate fmt --check clean on the changed file (opentofu/config.tm.hcl is pre-existing drift, untouched here)
Billable resources after teardown 0 clusters, 0 instances, 0 routers; only GCP's auto-created default VPC
IAM after teardown 0 custom roles, 0 principal:// bindings

Not covered

Neither defect is reachable by CI — both live in the Terramate deploy/destroy path, which nothing in the pipeline executes. This PR does not change that. A smoke test that runs the destroy workflow against a throwaway project would, and is worth considering separately given this is the second time the strand-a-cluster failure has appeared.

Smana added 2 commits August 24, 2026 17:37
`terramate script run deploy` runs `trivy config --exit-code=1` before apply, and
it failed on GCP-0050 ("Cluster does not override the default service account"),
so stage 1 never planned a resource.

It is a false positive of the same shape as GCP-0057, already documented two
lines above. The node pools take `local.service_account`, which the module
computes conditionally over `create_service_account`, so it renders as
`(known after apply)` and trivy assumes the default compute SA.

We do not override `create_service_account`, which defaults to true, so the
module creates a DEDICATED account. Confirmed in the plan rather than read from
the module source alone: `google_service_account.cluster_service_account[0]` is
created, carrying only metric_writer, node_service_account and
resource_metadata_writer bindings — least privilege, not Editor.

I could not establish WHEN this started firing; the stack deployed on 2026-08-23
and the module version is unchanged at 44.3.0. Recording that as unknown rather
than inventing a cause. The suppression stands on the plan evidence, not on a
theory about the regression.

The directive goes inside the contiguous `#trivy:ignore:` run — the comment
block above already warns that a prose line between the directives and `module`
silently voids them.
`terramate script run --reverse destroy` from opentofu/gcp/ visits gke/configure
BEFORE gke/init. That stack's destroy ran a bare `tofu destroy` under
`set -euo pipefail`, so any failure aborted the whole run before the billable
resource was touched — leaving a live GKE cluster with no workflow path to
remove it.

Hit for real on 2026-08-24, ~63 minutes after the cluster was created:

  Error: flux-system/gke-gcp-mycluster-0-vars failed to delete kubernetes
         resource: Unauthorized

The helm and kubectl providers hold a GCP access token acquired at plan time and
it had expired. Teardown stopped there; `gcloud container clusters list` showed
the cluster and both static nodes still RUNNING afterwards. It took a second,
manual run of gke/init's own destroy to remove them.

The expired token is the mild version. The usual reasons are worse — a broken
cluster, or a private endpoint unreachable because the tailnet is down — which
is exactly when a teardown matters most.

gke/init already solved this. Its destroy calls scripts/gke-destroy-stage2.sh in
`attempt` mode (graceful, never fails the caller) and defers state cleanup to a
`reconcile` job that runs only after the cluster is provably gone. That fix
landed on gke/init's own destroy and never reached the --reverse path, so the
hole reopened through a different door. This points configure's destroy at the
same helper, so the two entry points cannot drift apart again.

State is deliberately NOT cleared here: at this point the cluster may still
exist, so state may still be accurate. gke/init's reconcile job drops what is
left, and only afterwards.

Verified on the same teardown: with gke/init's tolerant path the cluster,
node pool and all IAM resources were destroyed, ending `[ok] stage 2 state
cleared` after reconciling 23 stale instances out of state.
@github-actions

Copy link
Copy Markdown
Contributor

🔍 Rendered manifest diff — this PR vs main (desired state)

No changes to the rendered desired state. ✅

@Smana
Smana merged commit 02bcdff into main Aug 24, 2026
8 checks passed
@Smana
Smana deleted the worktree-gcp-deploy-fixes branch August 24, 2026 15:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant