feat(scale-down): opt-in idle confirmation window before terminating not-busy runners

[jensenbox]

Problem

Closes the failure mode reported in #5085, including the harder variant discussed in its comments: GitHub's runner busy flag is not a reliable input for termination decisions. We traced 14 terminated-mid-job runners across one incident window (org-level, non-ephemeral, on-demand, module v7.10.0, scale-down on the default 5-minute schedule) and every one was killed through removeRunner's normal path on a busy: false reading, in two distinct shapes:

Variant A — assignment lag. The flag reads false for 25–60+ seconds after a job is already running on the runner:

job started 23:44:18 ... Busy: false at 23:45:18  → terminated → job killed
job started 23:44:23 ... Busy: false at 23:45:15  → terminated → job killed

Variant B — stale flag mid-job. The flag flips to false on a runner that has been continuously executing one job for many minutes:

03:05:10  Runner 'i-0a014...' - Busy: true   → correctly skipped
03:10:12  Runner 'i-0a014...' - Busy: false  → terminated; job died at 98% complete, all tests passing

Another case read false 12m15s into a running job. Because the DELETE call also succeeds in these cases (GitHub's server-side view is consistent with the wrong flag), no re-check-after-deregister sequencing can prevent Variant B — the module needs evidence across time instead of a single reading. With scale-to-zero config (idle_config = []) every instance past minimum_running_time_in_minutes is evaluated on every tick, so one stale-flag window kills several runners in the same invocation — we measured roughly one multi-runner burst per hour under CI load, each costing a full matrix rerun.

Change

Adds an opt-in confirmation window, default off (scale_down_idle_confirmation_seconds = 0 keeps today's single-reading behaviour bit-for-bit):

• On a not-busy reading, scale-down tags the instance ghr:idle_detected_at=<ISO8601> and defers termination.
• It terminates only when not-busy readings span at least the configured window (set it to one scale-down schedule interval to require two consecutive not-busy evaluations).
• Any busy reading clears the tag and resets the window — this is what saves the Variant B runners, which read true on the tick before their fatal stale reading.
• An unparsable tag value restarts the window rather than blocking or terminating.
• Each invocation logs a census (evaluated N runner(s): busy=… idle-deferred=… terminated=…) so "ran and found nothing" is distinguishable from "never ran".

No new IAM: the scale-down role already carries scoped ec2:CreateTags/DeleteTags for ghr:orphan. Threaded through the root module and multi-runner (runner_config.scale_down_idle_confirmation_seconds). Existing behaviours unchanged: bypass-removal, orphan handling, and the "only terminate EC2 after successful de-registration" guard.

Cost trade-off: a genuinely idle runner lives one extra evaluation interval before reap (~5 minutes on the default schedule).

Validation

• Unit tests: 5 new cases (first-reading deferral, in-window deferral, window-elapsed termination, busy-reading reset, unparsable-tag restart); full control-plane suite green on main (542 tests), eslint + prettier clean.
• Production: we have run this patch on our fleet (both pools, 300s window, 5-minute schedule) since 2026-07-28 05:32 UTC. Prior baseline: ~1 kill burst/hour. Since deploy: zero mid-job terminations in 14 hours of CI load, ~35 terminations all via the confirmed-idle path, zero de-registration failures. We directly observed the mechanism catching a would-have-been kill: a runner deferred on a false reading reported busy: true five minutes later and survived.

Happy to adjust naming/defaults or split the census logging out if you'd prefer a narrower diff. cc @npwolf / @ben-smyth from #5085 — this covers Variant B from your reports, which the deregister-then-recheck proposal cannot.

[edersonbrilhante]

@jensenbox Can you rebase? Heads up, I am working to refactor the module to introduce plugin layout. Ref: #5234

[jensenbox]

Rebased onto current main and ported to the compute-provider plugin framework from #5234. Sorry for the delay — the refactor landed the day after your rebase request, so this needed a port rather than a rebase.

Not a breaking interface change

The feature needs to persist "when was this runner first seen idle" across scale-down invocations. Before #5234 it tagged EC2 directly from the orchestration code, which is no longer the right layer.

ScaleDownComputeProvider now has markIdle / unmarkIdle, and RunnerInfo carries idleDetectedAt. Both methods are optional, deliberately:

• a provider with nowhere to persist per-runner state stays type-valid against the interface — nothing existing has to change
• scale-down checks for the capability and, when it is absent, skips the confirmation window entirely and keeps the previous single-reading behaviour rather than failing or deferring forever
• only a provider implementing them opts into the feature

aws/ec2 implements both with instance tags (ghr:idle_detected_at) — the same mechanism ghr:orphan already uses, so no new state store. The scaffold in templates/provider documents both as optional so plugin authors see them.

If you would rather express this as a separate capability interface than two optional methods on ScaleDownComputeProvider, say the word — that is a small change and I would rather match your intent for the framework than guess.

Why the feature

GitHub's busy flag can be stale. It reads false for runners actively executing a job, both shortly after assignment (25-60s lag) and deep into a running job (12+ minutes) — see #5085. A single busy=false reading is not sufficient evidence a runner is idle, so scale-down can terminate mid-job.

SCALE_DOWN_IDLE_CONFIRMATION_SECONDS (default 0, i.e. current behaviour) requires busy=false readings spanning at least that window before terminating. Any busy=true reading clears the marker and restarts the window.

Verification

• 265 passed across the full scale-runners suite, 7 files
• control-plane build clean, 0 TypeScript errors
• 5 new tests: window start, deferral mid-window, elapse-then-terminate, the disabled (0) path, and a provider implementing neither method
• I checked the new tests actually bite rather than assuming: stubbing idleConfirmed to always confirm fails exactly the window-start and deferral cases

Terraform plumbing (scale_down_idle_confirmation_seconds) is unchanged from the original PR and carried through the rebase cleanly.

[lochlanbennettodlum-taktile]

Opened #5397 with this change rebased onto current main and ported through the multi-runner translation layers, plus the three lint fixes. Kept you as commit author, all credit to you for the approach and the fleet data. Only real addition is clearing the idle tag for runners kept warm by idle_config, since those never hit the removal path. Happy to close mine if you'd rather push the rebase here instead.