fix: retry transient GitHub API failures when minting the runner token

[cewert]

What / why

A single transient GitHub API 503 took the runner down for ~23h (outage started 2026-07-19 20:07:43 EDT; Healthchecks correctly alerted).

Failure chain: registrar's curl -fsS ... | jq -re pipes an empty body into jq on the 503 → jq exits 4 → registrar container dies → runner never starts (depends_on: service_healthy) → --abort-on-container-exit → systemd restart storm → StartLimitBurst=5 parks the unit → no self-recovery. A momentary upstream blip became a multi-hour outage.

The fix

Wrap both GitHub API calls (installation access token, registration token) in a bounded retry helper:

• Retry curl-level failures + HTTP 429/5xx with backoff (3s, 6s, 9s).
• Fail fast on a genuine 4xx — bad creds/permissions won't recover on retry.
• Capture + log the real HTTP status, so a failure reads as transient HTTP 503 instead of an opaque jq exit 4.

Scope / non-goals

• Script only. No compose or systemd changes.
• RestartSec stays 5s (it also governs the normal ephemeral job-cycle — raising it would add latency to every job pickup).
• Retry budget (~21s + fast 5xx responses) stays well under the registrar healthcheck's ~90s window, so a slow mint is never marked unhealthy.
• StartLimitBurst=5 is intentionally left to still park on genuinely permanent faults (deprecated binary, revoked creds) and alert.

Testing

Shell test double run under dash and busybox ash (Alpine prod shell), all passing:

Scenario	Expected	Result
503, 503, 200 (this outage)	retries, succeeds on attempt 3	✅
401 bad creds	fail fast, 1 call	✅
persistent 500	exhaust 4 attempts, fail	✅
curl network error	treated as transient 000, retried	✅

Syntax validated under sh -n and dash -n.

Deploy note

Not yet deployed. Live service was already recovered via reset-failed + start and is draining the queued-job backlog. Deploy (ansible github_runner_ref bump) should wait until the runner is idle, since a restart cancels an in-flight ephemeral job.