diff --git a/.env.example b/.env.example index 14f8b8f..9ab55f2 100644 --- a/.env.example +++ b/.env.example @@ -37,10 +37,12 @@ TIMEZONE=America/New_York # ============================================================================= # Healthchecks.io push URL for runner liveness. If set, runner-entrypoint.sh -# pings /start at boot and every 60s while the container is alive. -# Configure the HC check with period=1m and grace=5m so brief gaps between -# ephemeral job cycles don't false-alert. Wire HC's integrations (Gotify, -# email, etc.) to alert on missed pings. Leave unset to disable. +# pings every 60s while the container is alive. Configure the HC check +# with period=1m and grace=20m: the runner is ephemeral (restarts between every +# job), so a tight grace false-alerts on normal restart gaps and trains you to +# ignore it. A generous grace makes a DOWN mean a sustained outage. Wire HC's +# integrations (Gotify, email, etc.) to alert on missed pings. Leave unset to +# disable. # # Why this exists: a deprecated runner binary, a crash loop, or a hung # container won't emit pings — HC.io alerts after the grace period. diff --git a/README.md b/README.md index 42509fa..361c2e1 100644 --- a/README.md +++ b/README.md @@ -295,7 +295,7 @@ The runner has three layers of failure detection, each catching a different clas ### Setting up the Healthchecks.io heartbeat 1. Create a check at (or your self-hosted HC instance) -2. Configure: **period** = `1 minute`, **grace** = `5 minutes`. The grace covers brief gaps between ephemeral job cycles. +2. Configure: **period** = `1 minute`, **grace** = `20 minutes`. The runner is ephemeral — the container restarts between every job — so the heartbeat has a gap on each cycle. A generous grace keeps those normal restart gaps (and brief self-recovering hiccups) from false-alerting, so a DOWN means a *sustained* outage worth paging on. (A too-tight grace causes alert fatigue: this check flipped 20× in 50 days on `grace=5m`, mostly self-recovered blips, which trained the alert to be ignored right before a real 18h outage.) 3. Add HC's **Gotify** (or email, Slack, etc.) integration to the check 4. Copy the check's ping URL into `.env`: @@ -305,7 +305,7 @@ The runner has three layers of failure detection, each catching a different clas 5. `sudo systemctl restart ` to pick up the change -The runner pings `/start` at container boot and `` every 60 s while it's alive. A deprecated runner that crashes within 10 s of startup never gets to the periodic ping, so HC.io alerts after the 5-min grace. +The runner pings `` every 60 s while it's alive. A runner that crashes within seconds of startup (deprecated binary, bad config, missing token) never gets to the periodic ping, so once the gap exceeds period + grace HC.io alerts. It does **not** ping `/start`: the container is ephemeral and dies before a run "completes", so a start signal never gets a matching success — it only adds noise. ## Troubleshooting diff --git a/runner-entrypoint.sh b/runner-entrypoint.sh index 03061d7..1379d3a 100755 --- a/runner-entrypoint.sh +++ b/runner-entrypoint.sh @@ -22,13 +22,17 @@ fi export RUNNER_TOKEN # Optional liveness heartbeat to Healthchecks.io. -# When HEALTHCHECKS_URL is set, ping /start at boot and every 60 s -# while this container is alive. Configure the HC check with period=1m and -# grace=5m so brief gaps between ephemeral job cycles don't false-alert. +# When HEALTHCHECKS_URL is set, ping every 60 s while this container is +# alive. Configure the HC check with a grace period well above one job-cycle +# restart (~20m) so brief gaps between ephemeral job cycles don't false-alert. # The background subshell dies with PID 1 when /entrypoint.sh exits, so we # don't need explicit cleanup. +# +# We intentionally do NOT ping /start: the container is ephemeral and dies +# before a run "completes", so a start signal never gets a matching success and +# only adds noise (it never registered on the HC side anyway). The plain 60s +# success ping is the whole signal. if [ -n "${HEALTHCHECKS_URL:-}" ]; then - curl -fsS -m 10 -o /dev/null "$HEALTHCHECKS_URL/start" || true ( while sleep 60; do curl -fsS -m 10 -o /dev/null "$HEALTHCHECKS_URL" || true