ci: the self-hosted fleet restarts mid-run, killing all in-flight jobs and reporting them as failures

[avrabe]

Summary

The self-hosted runner fleet restarts mid-run, killing every in-flight job
across all labels within the same second. Twice confirmed, and each time it
presents as a set of unrelated red checks that look like real failures.

Evidence

2026-08-27, PR #861 — three jobs on three different runners:

Code Coverage                       01-10   04:54:38 -> 04:56:51
Mutation Testing (rivet-cli)        01-9    04:54:38 -> 04:56:41
Mutation Testing (rivet-core, PR)   01-6    04:54:38 -> 04:56:41

04:56:37Z ##[error]The runner has received a shutdown signal. This can happen
          when the runner service is stopped, or a manually started runner is
          canceled.
04:56:38Z ##[error]The operation was canceled.

All three had been green on the previous commit of the same PR.

2026-08-26, PR #855 — same message at 04:33:47Z, killing WASM seam build, Test and Proptest (extended) across two labels within one second.
All three passed on re-run.

Why it costs more than a re-run

The jobs report failure, not cancelled. The GitHub UI shows a red X with
no failed step, so the natural reading is "these tests broke". On #855 I
initially attributed it to duplicate pull_request runs and had to publish a
correction; on #861 the killed jobs were the mutation gates, which had caught
a genuine bug in that same PR two ticks earlier — so a red there is exactly the
signal one should not wave through.

Distinguishing it requires opening a job's log and finding the shutdown line,
because:

• the job conclusion is failure;
• there is no failed step (the steps end skipped);
• only the raw log carries the reason.

Complication for #839

I have been reading exit 143 + shutdown signal on Kani Proofs as
hosted-runner reclaim. These two events show the identical message coming from
the self-hosted fleet. The message alone does not identify which pool
restarted, so #839's attribution rests on an inference the evidence does not
support and should be re-checked.

What would help

1. Find out why the fleet restarts. Unattended upgrades, a systemd timer, a
   config-management run, or an autoscaler reclaim would all produce this. If it
   is scheduled, moving it outside working hours costs nothing.
2. Make the state legible without opening logs. A post-job step that
   detects the shutdown and annotates the job would let a reader tell "the
   runner went away" from "the test failed" at a glance.
3. Failing that, at least record it here so the next person diagnosing three
   simultaneous unrelated failures finds this issue first.

Related

• ci: move CI Gate and the zola smoke off GitHub-hosted runners #855 — first confirmed occurrence, with the correction I had to publish
• ci: Kani Proofs fails ~58% with exit 143 and has never passed on a push to main #839 — Kani flakiness, whose attribution this undermines
• 🚨 CI runner pool liveness alert #860 — hosted starvation; a different mode, listed here only to keep the
  four runner failure modes distinct

[avrabe]

Triage note (scheduled routine, first pass on this issue — filed 2026-08-27, no prior comments).

The body pins the failure mode with unusual precision: two confirmed reproductions (#861 on 2026-08-27T04:56:37Z; #855 on 2026-08-26T04:33:47Z), three different runners each, all killed within one second, all reporting failure rather than cancelled — a conclusion that reads as a real test failure and requires opening the raw log to falsify. The load-bearing observation is the note that "The message alone does not identify which pool restarted, so #839's attribution rests on an inference the evidence does not support and should be re-checked" — that undermines the "hosted-runner reclaim" branch of #839's Phase-0 diagnostic AC and needs to feed back there once diagnosed here.

The three "what would help" options in the body compose cleanly (they aren't alternatives): (1) diagnose why the fleet restarts, (2) make the state legible without opening logs, (3) at minimum record the pattern so the next person finds it. (2) is the fast, always-safe surface fix; (1) is the durable one; (3) is what filing this issue already accomplishes.

Sketch matching the kill-criterion shape used across adjacent triage threads (#854 / #839 / #838 / #833 / #832) — drop in / revise:

## Acceptance Criteria — Phase A (legibility: annotate the killed-by-restart state)

- [ ] Every workflow job that runs on the self-hosted fleet grows a
      final step (`if: always()`) that scans its job log for the exact
      `##[error]The runner has received a shutdown signal.` line and,
      if found, emits a job annotation + step-summary line explicitly
      naming the state: "runner reclaimed mid-job — this is not a test
      failure; see #862." The annotation surfaces in the PR checks view
      without opening a log. Detection runs on both `success` and
      `failure` outcomes (the shutdown can land in either).
- [ ] The same detection produces a machine-checkable marker: a
      `runner-reclaimed: true` line in the step summary (or a
      `::notice::` with a fixed prefix) so downstream tooling (a
      `rivet ci` triage helper or a bot) can distinguish reclaim from
      real failure without regex-scraping the log.
- [ ] Regression fixture: an integration test (or a documented
      copy-paste manual reproduction) that runs a self-hosted job and
      injects the shutdown-signal line into the log, and asserts that
      the annotation + step-summary marker appear. Purpose: keeps the
      annotation surface honest as the workflow evolves.

## Acceptance Criteria — Phase B (diagnose: pin why the fleet restarts)

- [ ] Enumerate the four candidates the body names — unattended
      upgrades, systemd timer, config-management run, autoscaler
      reclaim — and produce evidence for/against each on the fleet
      hosts: `journalctl --since '-24h' | grep -iE
      'reboot|shutdown|unattended|puppet|ansible|autoscaler'`,
      `systemctl list-timers`, `apt-config dump | grep -i unattended`,
      and the autoscaler's own logs if one exists. Record in a
      decision artifact under `artifacts/decisions/` naming the cause
      and the evidence trail.
- [ ] If the cause is scheduled (unattended upgrades / systemd timer /
      config-management), move the schedule outside working hours
      (naming a specific window in the decision artifact) and disable
      the specific trigger during CI-active windows where possible.
- [ ] If the cause is autoscaler reclaim, either raise the
      minimum-alive count so mid-job reclaim can't happen, or wire the
      autoscaler's drain signal into the runner service so in-flight
      jobs complete before shutdown (whichever fits the autoscaler's
      contract). Record the pick + rationale in the decision artifact.
- [ ] Follow-up comment on #839 corrects the "hosted-runner reclaim"
      attribution in that issue's diagnostic AC — the shutdown-signal
      line is not sufficient to attribute a kill to the hosted pool,
      because the self-hosted fleet emits the same message.

## Acceptance Criteria — cross-cutting

- [ ] `cargo test -p rivet-cli` green; `rivet validate` PASS on the
      rivet repo (Phase A is workflow-yaml + one shell/python detection
      script; Phase B is decision artifact + workflow config only).
- [ ] Commit trailers: Phase A is CI-only so `Trace: skip` is acceptable
      (preferred: `Refs: #862, #839, #855` for `rivet commits`
      linkability); Phase B carries `Refs: #862, #839` and, if a REQ is
      filed for "runner-reclaimed state must be legible in CI",
      `Implements: REQ-NNN`.
- [ ] **Explicitly out of scope (file separately):** any change to the
      hosted-runner reclaim path (#860's territory); rebuilding the
      self-hosted fleet on a different orchestrator; a `rivet ci`
      subcommand that queries the checks API and flags
      runner-reclaimed jobs (nice to have, own REQ).

Adjacent state worth flagging:

• ci: Kani Proofs fails ~58% with exit 143 and has never passed on a push to main #839 (Kani ~58% flakes, exit 143 + shutdown signal) — the body here explicitly notes that ci: Kani Proofs fails ~58% with exit 143 and has never passed on a push to main #839's Phase-0 diagnostic AC attributes the kill to hosted-runner reclaim. This issue's evidence undermines that attribution, so ci: Kani Proofs fails ~58% with exit 143 and has never passed on a push to main #839's diagnostic instrumentation should be widened to distinguish which pool restarted (hostname / self-hosted label in the annotation), or ci: Kani Proofs fails ~58% with exit 143 and has never passed on a push to main #839's Phase 0 should adopt this issue's Phase A annotation as a building block. Sequencing suggestion: Phase A here first, then re-open ci: Kani Proofs fails ~58% with exit 143 and has never passed on a push to main #839's diagnostic with the pool-identifying annotation in hand.
• ci: move CI Gate and the zola smoke off GitHub-hosted runners #855 — first confirmed occurrence; body already links back.
• 🚨 CI runner pool liveness alert #860 — hosted starvation; a distinct failure mode, body distinguishes it correctly.
• Same "silent state that reads as a real failure" family as ci: Test job silently falls back to cargo test, producing no JUnit XML and losing retries — three stacked silencers #833 (nextest fallback silently drops JUnit XML), serve: /api/v1/artifacts silently truncates at limit=1000 (repo has 1017) — no truncation signal in the response #832 (silent truncation at limit=1000), rivet verify advances on the existence of evidence, not its sufficiency #838 (verify advances on marker-only evidence), friction: a dead externals path: poisons all cross-repo resolution (no git: fallback), and validate then reports '0 broken cross-refs' beside dozens of unresolved-target errors #854 (dead externals silently reports 0 broken cross-refs). Phase A here is the direct analogue of serve: /api/v1/artifacts silently truncates at limit=1000 (repo has 1017) — no truncation signal in the response #832's truncated: true fix, moved from data-plane to CI-plane.

Process side-rail: consulted https://pulseengine.eu/blog/ at the start of this run — 200 OK. Nothing on the blog speaks to CI-annotation convention or per-issue AC shape; the kill-criterion convention used across adjacent triage threads is the working default.

Not implemented this run — no PR opened; the AC block on the body is the gating input. Once you pin whether Phase A comes first (fast, always-safe) or Phases A+B land in one branch, the next routine pass can open a focused branch.

---

Generated by Claude Code