From 95bb9cb9cc76ec73c0ad9711a6f3e0ed40ebcb89 Mon Sep 17 00:00:00 2001 From: Charles Ewert Date: Sat, 11 Jul 2026 17:30:39 -0400 Subject: [PATCH] heartbeat: drop non-functional /start ping, document 20m grace The runner-entrypoint pinged /start at container boot, but it never registered on the HC side (0 start pings in 71k) and can't work by design: the runner is ephemeral and the container dies before a run "completes", so a start signal never gets a matching success ping. It only added noise. Alerting relies solely on the plain 60s success ping, which is unaffected. Also update the recommended HC grace from 5m to 20m in the README/.env.example. A tight grace false-alerts on normal ephemeral-runner restart gaps: this check flipped 20x in 50 days on grace=5m (mostly self-recovered blips), which trained the Gotify alert to be ignored right before a real ~18h outage. A generous grace makes a DOWN mean a sustained outage worth paging on. --- .env.example | 10 ++++++---- README.md | 4 ++-- runner-entrypoint.sh | 12 ++++++++---- 3 files changed, 16 insertions(+), 10 deletions(-) diff --git a/.env.example b/.env.example index 14f8b8f..9ab55f2 100644 --- a/.env.example +++ b/.env.example @@ -37,10 +37,12 @@ TIMEZONE=America/New_York # ============================================================================= # Healthchecks.io push URL for runner liveness. If set, runner-entrypoint.sh -# pings /start at boot and every 60s while the container is alive. -# Configure the HC check with period=1m and grace=5m so brief gaps between -# ephemeral job cycles don't false-alert. Wire HC's integrations (Gotify, -# email, etc.) to alert on missed pings. Leave unset to disable. +# pings every 60s while the container is alive. Configure the HC check +# with period=1m and grace=20m: the runner is ephemeral (restarts between every +# job), so a tight grace false-alerts on normal restart gaps and trains you to +# ignore it. A generous grace makes a DOWN mean a sustained outage. Wire HC's +# integrations (Gotify, email, etc.) to alert on missed pings. Leave unset to +# disable. # # Why this exists: a deprecated runner binary, a crash loop, or a hung # container won't emit pings — HC.io alerts after the grace period. diff --git a/README.md b/README.md index 42509fa..361c2e1 100644 --- a/README.md +++ b/README.md @@ -295,7 +295,7 @@ The runner has three layers of failure detection, each catching a different clas ### Setting up the Healthchecks.io heartbeat 1. Create a check at (or your self-hosted HC instance) -2. Configure: **period** = `1 minute`, **grace** = `5 minutes`. The grace covers brief gaps between ephemeral job cycles. +2. Configure: **period** = `1 minute`, **grace** = `20 minutes`. The runner is ephemeral — the container restarts between every job — so the heartbeat has a gap on each cycle. A generous grace keeps those normal restart gaps (and brief self-recovering hiccups) from false-alerting, so a DOWN means a *sustained* outage worth paging on. (A too-tight grace causes alert fatigue: this check flipped 20× in 50 days on `grace=5m`, mostly self-recovered blips, which trained the alert to be ignored right before a real 18h outage.) 3. Add HC's **Gotify** (or email, Slack, etc.) integration to the check 4. Copy the check's ping URL into `.env`: @@ -305,7 +305,7 @@ The runner has three layers of failure detection, each catching a different clas 5. `sudo systemctl restart ` to pick up the change -The runner pings `/start` at container boot and `` every 60 s while it's alive. A deprecated runner that crashes within 10 s of startup never gets to the periodic ping, so HC.io alerts after the 5-min grace. +The runner pings `` every 60 s while it's alive. A runner that crashes within seconds of startup (deprecated binary, bad config, missing token) never gets to the periodic ping, so once the gap exceeds period + grace HC.io alerts. It does **not** ping `/start`: the container is ephemeral and dies before a run "completes", so a start signal never gets a matching success — it only adds noise. ## Troubleshooting diff --git a/runner-entrypoint.sh b/runner-entrypoint.sh index 03061d7..1379d3a 100755 --- a/runner-entrypoint.sh +++ b/runner-entrypoint.sh @@ -22,13 +22,17 @@ fi export RUNNER_TOKEN # Optional liveness heartbeat to Healthchecks.io. -# When HEALTHCHECKS_URL is set, ping /start at boot and every 60 s -# while this container is alive. Configure the HC check with period=1m and -# grace=5m so brief gaps between ephemeral job cycles don't false-alert. +# When HEALTHCHECKS_URL is set, ping every 60 s while this container is +# alive. Configure the HC check with a grace period well above one job-cycle +# restart (~20m) so brief gaps between ephemeral job cycles don't false-alert. # The background subshell dies with PID 1 when /entrypoint.sh exits, so we # don't need explicit cleanup. +# +# We intentionally do NOT ping /start: the container is ephemeral and dies +# before a run "completes", so a start signal never gets a matching success and +# only adds noise (it never registered on the HC side anyway). The plain 60s +# success ping is the whole signal. if [ -n "${HEALTHCHECKS_URL:-}" ]; then - curl -fsS -m 10 -o /dev/null "$HEALTHCHECKS_URL/start" || true ( while sleep 60; do curl -fsS -m 10 -o /dev/null "$HEALTHCHECKS_URL" || true