Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 6 additions & 4 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -37,10 +37,12 @@ TIMEZONE=America/New_York
# =============================================================================

# Healthchecks.io push URL for runner liveness. If set, runner-entrypoint.sh
# pings <URL>/start at boot and <URL> every 60s while the container is alive.
# Configure the HC check with period=1m and grace=5m so brief gaps between
# ephemeral job cycles don't false-alert. Wire HC's integrations (Gotify,
# email, etc.) to alert on missed pings. Leave unset to disable.
# pings <URL> every 60s while the container is alive. Configure the HC check
# with period=1m and grace=20m: the runner is ephemeral (restarts between every
# job), so a tight grace false-alerts on normal restart gaps and trains you to
# ignore it. A generous grace makes a DOWN mean a sustained outage. Wire HC's
# integrations (Gotify, email, etc.) to alert on missed pings. Leave unset to
# disable.
#
# Why this exists: a deprecated runner binary, a crash loop, or a hung
# container won't emit pings — HC.io alerts after the grace period.
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -295,7 +295,7 @@ The runner has three layers of failure detection, each catching a different clas
### Setting up the Healthchecks.io heartbeat

1. Create a check at <https://healthchecks.io/> (or your self-hosted HC instance)
2. Configure: **period** = `1 minute`, **grace** = `5 minutes`. The grace covers brief gaps between ephemeral job cycles.
2. Configure: **period** = `1 minute`, **grace** = `20 minutes`. The runner is ephemeral — the container restarts between every job — so the heartbeat has a gap on each cycle. A generous grace keeps those normal restart gaps (and brief self-recovering hiccups) from false-alerting, so a DOWN means a *sustained* outage worth paging on. (A too-tight grace causes alert fatigue: this check flipped 20× in 50 days on `grace=5m`, mostly self-recovered blips, which trained the alert to be ignored right before a real 18h outage.)
3. Add HC's **Gotify** (or email, Slack, etc.) integration to the check
4. Copy the check's ping URL into `.env`:

Expand All @@ -305,7 +305,7 @@ The runner has three layers of failure detection, each catching a different clas

5. `sudo systemctl restart <service-name>` to pick up the change

The runner pings `<URL>/start` at container boot and `<URL>` every 60 s while it's alive. A deprecated runner that crashes within 10 s of startup never gets to the periodic ping, so HC.io alerts after the 5-min grace.
The runner pings `<URL>` every 60 s while it's alive. A runner that crashes within seconds of startup (deprecated binary, bad config, missing token) never gets to the periodic ping, so once the gap exceeds period + grace HC.io alerts. It does **not** ping `<URL>/start`: the container is ephemeral and dies before a run "completes", so a start signal never gets a matching success — it only adds noise.

## Troubleshooting

Expand Down
12 changes: 8 additions & 4 deletions runner-entrypoint.sh
Original file line number Diff line number Diff line change
Expand Up @@ -22,13 +22,17 @@ fi
export RUNNER_TOKEN

# Optional liveness heartbeat to Healthchecks.io.
# When HEALTHCHECKS_URL is set, ping <URL>/start at boot and <URL> every 60 s
# while this container is alive. Configure the HC check with period=1m and
# grace=5m so brief gaps between ephemeral job cycles don't false-alert.
# When HEALTHCHECKS_URL is set, ping <URL> every 60 s while this container is
# alive. Configure the HC check with a grace period well above one job-cycle
# restart (~20m) so brief gaps between ephemeral job cycles don't false-alert.
# The background subshell dies with PID 1 when /entrypoint.sh exits, so we
# don't need explicit cleanup.
#
# We intentionally do NOT ping <URL>/start: the container is ephemeral and dies
# before a run "completes", so a start signal never gets a matching success and
# only adds noise (it never registered on the HC side anyway). The plain 60s
# success ping is the whole signal.
if [ -n "${HEALTHCHECKS_URL:-}" ]; then
curl -fsS -m 10 -o /dev/null "$HEALTHCHECKS_URL/start" || true
(
while sleep 60; do
curl -fsS -m 10 -o /dev/null "$HEALTHCHECKS_URL" || true
Expand Down