fix(gcp): unblock the deploy, and stop --reverse destroy stranding a cluster - #1826
Merged
Conversation
`terramate script run deploy` runs `trivy config --exit-code=1` before apply, and
it failed on GCP-0050 ("Cluster does not override the default service account"),
so stage 1 never planned a resource.
It is a false positive of the same shape as GCP-0057, already documented two
lines above. The node pools take `local.service_account`, which the module
computes conditionally over `create_service_account`, so it renders as
`(known after apply)` and trivy assumes the default compute SA.
We do not override `create_service_account`, which defaults to true, so the
module creates a DEDICATED account. Confirmed in the plan rather than read from
the module source alone: `google_service_account.cluster_service_account[0]` is
created, carrying only metric_writer, node_service_account and
resource_metadata_writer bindings — least privilege, not Editor.
I could not establish WHEN this started firing; the stack deployed on 2026-08-23
and the module version is unchanged at 44.3.0. Recording that as unknown rather
than inventing a cause. The suppression stands on the plan evidence, not on a
theory about the regression.
The directive goes inside the contiguous `#trivy:ignore:` run — the comment
block above already warns that a prose line between the directives and `module`
silently voids them.
`terramate script run --reverse destroy` from opentofu/gcp/ visits gke/configure
BEFORE gke/init. That stack's destroy ran a bare `tofu destroy` under
`set -euo pipefail`, so any failure aborted the whole run before the billable
resource was touched — leaving a live GKE cluster with no workflow path to
remove it.
Hit for real on 2026-08-24, ~63 minutes after the cluster was created:
Error: flux-system/gke-gcp-mycluster-0-vars failed to delete kubernetes
resource: Unauthorized
The helm and kubectl providers hold a GCP access token acquired at plan time and
it had expired. Teardown stopped there; `gcloud container clusters list` showed
the cluster and both static nodes still RUNNING afterwards. It took a second,
manual run of gke/init's own destroy to remove them.
The expired token is the mild version. The usual reasons are worse — a broken
cluster, or a private endpoint unreachable because the tailnet is down — which
is exactly when a teardown matters most.
gke/init already solved this. Its destroy calls scripts/gke-destroy-stage2.sh in
`attempt` mode (graceful, never fails the caller) and defers state cleanup to a
`reconcile` job that runs only after the cluster is provably gone. That fix
landed on gke/init's own destroy and never reached the --reverse path, so the
hole reopened through a different door. This points configure's destroy at the
same helper, so the two entry points cannot drift apart again.
State is deliberately NOT cleared here: at this point the cluster may still
exist, so state may still be accurate. gke/init's reconcile job drops what is
left, and only afterwards.
Verified on the same teardown: with gke/init's tolerant path the cluster,
node pool and all IAM resources were destroyed, ending `[ok] stage 2 state
cleared` after reconciling 23 stale instances out of state.
Contributor
🔍 Rendered manifest diff — this PR vs
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two workflow defects found by actually running the GCP deploy/verify/teardown cycle today. Both block the documented path; neither is reachable by CI.
1.
trivy configblocked the deploy before a single resource was plannedterramate script run deployrunstrivy config --exit-code=1ahead of apply, and it failed on GCP-0050 ("Cluster does not override the default service account"). Stage 1 never got to plan.It is a false positive of the same shape as GCP-0057, already documented two lines above it in the same file. Node pools take
local.service_account, which the module computes conditionally overcreate_service_account, so it renders as(known after apply)and trivy assumes the default compute SA.Verified in the plan rather than read from the module source alone:
A dedicated account with three narrowly-scoped bindings — not Editor, and not the default compute SA.
I could not establish when this started firing: the stack deployed cleanly on 2026-08-23 and the module version is unchanged at 44.3.0. That is recorded as unknown in the commit rather than dressed up with a theory. The suppression stands on the plan evidence.
2.
--reverse destroystranded a live clusterterramate script run --reverse destroyfromopentofu/gcp/visitsgke/configurebeforegke/init. That stack's destroy ran a baretofu destroyunderset -euo pipefail, so a failure aborted the run before the billable resource was touched.Hit for real today, ~63 minutes after cluster creation:
The helm and kubectl providers hold a GCP access token acquired at plan time; it had expired. Teardown stopped there, and
gcloud container clusters listshowed the cluster and both static nodes still RUNNING. Removing them took a second, manual run ofgke/init's destroy.The expired token is the mild version. The usual reasons are worse — a broken cluster, or a private endpoint unreachable because the tailnet is down — which is exactly when a teardown matters most.
gke/initalready solved this. Its destroy callsscripts/gke-destroy-stage2.shinattemptmode (graceful, never fails the caller) and defers state cleanup to areconcilejob that runs only once the cluster is provably gone. That fix landed ongke/init's own destroy and never reached the--reversepath, so the hole reopened through a different door. This pointsconfigure's destroy at the same helper, so the two entry points cannot drift apart again.State is deliberately not cleared here: at this point the cluster may still exist, so state may still be accurate.
gke/init'sreconcilejob drops what is left, and only afterwards.Evidence
Both fixes were exercised end to end on a real cluster today, then the cluster was destroyed.
Apply complete! Resources: 21 added, 0 changed, 0 destroyed— cluster, Cilium, Flux Operator, Flux Instance[ok] stage 2 state clearedafter reconciling 23 stale instances out of stateterramate fmt --checkopentofu/config.tm.hclis pre-existing drift, untouched here)defaultVPCprincipal://bindingsNot covered
Neither defect is reachable by CI — both live in the Terramate deploy/destroy path, which nothing in the pipeline executes. This PR does not change that. A smoke test that runs the destroy workflow against a throwaway project would, and is worth considering separately given this is the second time the strand-a-cluster failure has appeared.