fix(JDWLABS-465): retry porkbun-ddns across pods on transient DNS failures

[jdwillmsen]

Problem

porkbun-ddns-29799945 failed with curl: (6) Could not resolve host: api.porkbun.com (all 4 in-pod attempts, 09:45:01-09:45:36 UTC) and fired KubeJobFailed. With backoffLimit: 0 a single pod's resolver outage is a failed Job.

Investigation: transient, not node-specific

• Failure history (kube-state-metrics, 14d): 8 of ~1140 runs failed, all BackoffLimitExceeded. Four on talos-lx0-6a4 (Aug 18-21), four on talos-4h8-zy6 (Aug 26-29) -- failures follow wherever the scheduler happens to place the job, not one node. Every failed run was followed by a successful one 15 minutes later on the same node (~0.7% failure rate).
• The in-pod curl retry already merged (--retry 3 --retry-delay 5) was live for the last two failures; the blip simply outlasted its ~35s window.
• CoreDNS was healthy during the window (Prometheus 09:35-09:55 UTC): both pods up=1, steady ~4 req/s each, zero SERVFAIL, zero coredns_forward_healthcheck_broken_total, no errors-plugin log lines in 24h, 0 restarts. Neither CoreDNS pod runs on talos-4h8-zy6.
• Node was fine: talos-4h8-zy6 Ready, kubelet targets up, CPU ~3%, no node events. Other pods on the node (argocd application-controller) logged no lookup errors at the time.
• No CoreDNS*/KubeDNS*/Cilium* alerts fired in 14d.

Conclusion: sporadic, short (35s < t < 15min) resolver blips seen only by individual freshly-created pods, on multiple nodes. CoreDNS itself is not implicated. A likely contributor is the CNI state on worker nodes (both kube-flannel and cilium DaemonSets are present, mid-migration; cilium_* metrics are not scraped so drop counters could not be checked) -- noted below as a follow-up, not addressed here.

Fix

tenants/platform/services/porkbun-ddns/postInstall/cronjob.yaml:

• backoffLimit: 0 -> 2 -- a fresh pod (new sandbox/endpoint) is the retry that matches the observed failure shape.
• curl --retry 3 -> --retry 5 (--retry-all-errors --retry-delay 5 and -4 unchanged): ~65s per pod attempt.
• activeDeadlineSeconds: 600 so pod retries plus controller backoff can never spill into the next 15-minute slot (concurrencyPolicy: Forbid already prevents overlap).
• Comments rewritten to record the evidence rather than the earlier "no pod retries" rationale.

An outage that outlasts all three pods still fails the Job and pages, which remains the case worth surfacing.

Evidence / validation

• yamllint clean, kubeconform -strict valid
• Inline script extracted and passed sh -n and shellcheck -s sh
• tools/check-image-pins.py, tools/check-orphaned-manifests.py clean

Follow-up (out of scope)

• Investigate why fresh pods intermittently lose DNS for 30s+ on worker nodes: flannel + cilium coexistence, and get cilium_drop_count_total scraped so drops are observable.
• Same failure mode probably affects other short-lived Job pods (KubeJobFailed also fired in ai-sre, jdwillmsen-prd, vault over 14d; not inspected).

🤖 Generated with Claude Code

https://claude.ai/code/session_01B6FWEmAs2beQvKnX4LwWN2