From b5daccaa4e7e415d5bbaf6fa82f2e73f7624f8e7 Mon Sep 17 00:00:00 2001 From: Charles Ewert Date: Mon, 20 Jul 2026 19:56:25 -0400 Subject: [PATCH] fix: retry transient GitHub API failures when minting the runner token A single transient GitHub API 503 took the runner down for ~23h (2026-07-19 20:07 EDT). The registrar's `curl -fsS ... | jq -re` piped an empty body into jq on the 503, jq exited 4, the registrar container died, the runner never started (gated on service_healthy), and systemd's StartLimitBurst=5 parked the whole unit until manual recovery. A momentary upstream blip became a multi-hour outage with no self-recovery. Wrap both GitHub API calls (installation access token, registration token) in a bounded retry with backoff: - retry on curl-level failures and HTTP 429/5xx (3s, 6s, 9s backoff), - fail fast on a genuine 4xx (bad creds/permissions won't fix on retry), - capture and log the real HTTP status, so a failure reads as "transient HTTP 503" instead of an opaque jq exit 4. Retry budget (~21s of sleeps + fast 5xx responses) stays well under the registrar healthcheck's ~90s window, so a slow mint is never marked unhealthy. No compose or systemd changes; RestartSec stays at 5s to keep the normal ephemeral job-cycle fast. Verified with a shell test double under dash and busybox ash covering: 503->503->200 (recovers), 401 (fails fast, 1 call), persistent 500 (exhausts 4 attempts), and curl network error (treated as transient). --- mint-runner-token.sh | 67 ++++++++++++++++++++++++++++++++++++-------- 1 file changed, 55 insertions(+), 12 deletions(-) diff --git a/mint-runner-token.sh b/mint-runner-token.sh index b176f6c..e85f49c 100755 --- a/mint-runner-token.sh +++ b/mint-runner-token.sh @@ -56,6 +56,53 @@ esac NOW=$(date +%s) EXP=$((NOW + 540)) # 9 min; GitHub max is 10 +# POST to a GitHub API endpoint, tolerating transient failures. +# +# Why this exists: GitHub's API returns routine, short-lived 5xx/429s, and a +# single one used to be fatal. The old `curl -fsS ... | jq` piped an empty body +# into jq on any HTTP error, so the registrar died with an opaque `jq` exit 4; +# the runner then never started and systemd's StartLimitBurst parked the whole +# unit until a human intervened. A momentary blip became a multi-hour outage. +# +# So: retry on network errors and HTTP 429/5xx with backoff, fail fast on a +# genuine 4xx (bad creds / permissions — retrying won't help), and log the real +# HTTP status either way. Retry budget (~21s of sleeps) stays well under the +# registrar healthcheck's ~90s window so a slow mint can't be marked unhealthy. +# +# Args: