CI capacity: 56 of 63 job definitions target ubuntu-latest while 12 self-hosted runners idle — 401 jobs queued, 0 in progress during the v0.59 merge wave

[avrabe]

The measurement

During the v0.59 merge wave (8 PRs in flight) CI reached a hard stall:

2026-08-26T02:20Z   completed=34   in_progress=0   queued=401
oldest queued job waiting ~52 min; two samples 20 min apart both showed
in_progress=0 with completed unchanged at 34.

Meanwhile the org's self-hosted fleet was 12 runners online, every one
busy=false.

That is not a misconfiguration — I checked, and the labels match:

requested by workflows	runners satisfying it
[self-hosted, linux, x64, rust-cpu]	7 of 12
[self-hosted, linux, x64, light]	1 of 12

The fleet is schedulable. It is simply almost never asked for:

runs-on: ubuntu-latest                        56 job definitions
runs-on: [self-hosted, linux, x64, rust-cpu]   3
runs-on: [self-hosted, linux, x64, light]      2
runs-on: macos-latest                          1

56 of 63 job definitions target GitHub-hosted runners. synth runs ~50 jobs
per PR (the oracle sweep is the bulk of it), so N PRs in flight queue ~50N jobs
against GitHub-hosted concurrency shared with the rest of the org — gale and loom
were queued at the same moment — while 12 paid runners idle.

Why it matters beyond waiting

The hub model this repo uses (parallel lanes, PR-not-merge, coordinator gates)
structurally produces bursts of 6–8 simultaneous PRs. At ~50 jobs each that is
a 300–400 job burst every wave. The stall is therefore not a one-off; it is the
expected steady state of the working model, and it currently gates every merge
and therefore every release cut.

What I did NOT do, and why

I did not move jobs to the self-hosted fleet. Under this repo's branch
protection a required check that cannot run deadlocks all merges — so
retargeting any of the 9 required contexts (Format, Clippy, Test,
Z3 Verification, Claim Check, Version Pin Sweep, Bazel Build & Proofs,
Kani Verification, Rivet Validation) is a change to gate infrastructure and
should be deliberate, not a side effect of clearing a queue.

I did cancel three superseded runs (older SHAs on branches that had been
force-pushed, including a deliberate red-first probe whose RED result is already
recorded in #1060's body). Verified surgical: current-head rollups unchanged and
zero required checks left CANCELLED on any of the 8 open PRs.

Options, roughly in increasing order of commitment

1. Move the non-required oracle sweep to rust-cpu. The ~40 scripts/repro/*
   execution differentials are the bulk of the burst and none of them are in the
   required set, so a mis-scheduled one blocks nothing. Largest relief, lowest
   risk.
2. Stagger lane pushes so a wave lands as 2–3 PRs rather than 8. Cheap, but
   it slows the hub and does not use the idle capacity.
3. Retarget some required checks to self-hosted. Highest relief, highest
   risk — needs the deadlock caveat above respected, and probably a period of
   running both before flipping the required context.
4. More GitHub-hosted concurrency. Spend, not engineering.

Option 1 looks like the right first move: it is reversible, it touches nothing
required, and it aims exactly at the jobs that make the burst large.

Reproduce

gh api orgs/pulseengine/actions/runners --jq '.runners[]|"\(.name) \(.labels|map(.name)|join(","))"'
grep -hoE '^\s+runs-on:.*' .github/workflows/*.yml | sed 's/^\s*runs-on:\s*//' | sort | uniq -c | sort -rn

Note for whoever measures the queue: gh run list --json status reports run
status, and a run stays queued while its jobs complete. Census at job level
(actions/runs/<id>/jobs) or the number is misleading.

[avrabe]

[synth maintainer loop] — the stall reproduced live, with a full per-runner census this time.

Measured just now while four v0.60 PRs sat blocked:

repos/pulseengine/synth runs:   in_progress = 0     queued = 8
orgs/pulseengine runners:       12 online, 4 busy, 8 IDLE

Zero synth jobs executing. Eight idle runners. Per-runner labels — the census, not a generalization from one box (that mistake was made here before):

pulseengine-ci-01-1    busy   lean-mem
pulseengine-ci-01-2    busy   lean-mem
pulseengine-ci-01-3    busy   lean-mem
pulseengine-ci-01-4    busy   lean-mem
pulseengine-ci-01-5    IDLE   rust-cpu
pulseengine-ci-01-6    IDLE   rust-cpu
pulseengine-ci-01-7    IDLE   rust-cpu
pulseengine-ci-01-8    IDLE   light
pulseengine-ci-01-9    IDLE   rust-cpu, podman
pulseengine-ci-01-10   IDLE   rust-cpu
pulseengine-ci-01-11   IDLE   rust-cpu
pulseengine-ci-01-12   IDLE   rust-cpu, netns

The four busy ones are serving other repos — org-level runners are shared. So the idle capacity is real and synth-available; the queued synth jobs simply do not request it, because they name ubuntu-latest.

Every one of the four blocked PRs is stuck on the same six required contexts — Clippy, Test, Z3 Verification, Claim Check, Bazel Build & Proofs, Kani Verification — while Format, Version Pin Sweep and Rivet Validation (the cheap three) had already gone green. The heavy jobs are exactly the ones the rust-cpu runners exist for.

Two pieces of queue residue found while diagnosing

• Run 33080182202 was a superseded CI run for feat/artifact-split-per-requirement-1059 at a5140957, still queued and consuming a concurrency slot although 17b8101e had its own queued run. Cancelled; verified 0 cancelled check-runs on the PR's current head afterwards, so no required context was orphaned.
• Run 32219153827 (feat/ship-verify-feature-1000, PR feat(#1000): ship 'synth verify' in released artifacts + non-vacuity smoke gate (RQ-58-SHIPVERIFY) #1002, merged 2026-08-19) has been queued for eight days and cannot be cancelled: HTTP 409 — Cannot cancel a workflow run that has not been queued yet. A run stuck in a pre-queued state that no API call can clear is its own finding; it is not obvious it consumes a slot, but it is certainly not doing anything useful.

What this changes about the fix

The issue already names the shape (56 of 63 job definitions target ubuntu-latest). The census adds the constraint that matters for how to fix it: 7 of the 8 idle runners satisfy rust-cpu and exactly one satisfies light — so a blanket retarget would pile every job onto one class and starve differently.

The care that is non-negotiable: a required context that names a label no online runner satisfies does not fail, it never runs — and a required check that never runs deadlocks every merge in the repo. So retarget non-required jobs first and watch one full cycle, and move a required context only one at a time with its own green cycle before the next.

[avrabe]

Follow-up: the natural experiment already ran, on four PRs simultaneously, and it isolates the cause exactly.

Of the 9 required contexts, three run on self-hosted and six on GitHub-hosted:

Format                 -> [self-hosted, linux, x64, light]
Version Pin Sweep      -> [self-hosted, linux, x64, light]
Rivet Validation       -> [self-hosted, linux, x64, rust-cpu]
Clippy                 -> ubuntu-latest
Test                   -> ubuntu-latest
Z3 Verification        -> ubuntu-latest
Claim Check            -> ubuntu-latest
Bazel Build & Proofs   -> ubuntu-latest
Kani Verification      -> ubuntu-latest

Across four PRs opened within ~30 minutes of each other (#1084, #1086, #1088, #1089), every PR reached exactly 3/9 within minutes and then stopped. The three that completed are the three self-hosted ones — on every PR, without exception. The six that stalled are the six ubuntu-latest ones, on every PR.

Same commits, same moment, same queue. The split is not correlated with job cost, language, or cache state — Format and Rivet Validation are not trivially cheaper than Clippy. It is entirely explained by which pool the job asks for.

Full runs-on census across all workflows, confirming the issue's original count:

  56  ubuntu-latest
   4  [self-hosted, linux, x64, rust-cpu]
   2  [self-hosted, linux, x64, light]
   1  macos-latest
   1  ${{ matrix.os }}

What this makes concrete about the fix

The six required ubuntu-latest jobs are the release-blocking set, and rust-cpu — where 7 of the 8 idle runners live — is the right pool for at least Clippy, Test and Kani Verification (plain Rust toolchain work; Rivet Validation already runs there successfully, so the pool is proven for cargo-shaped jobs).

The two that need thought rather than a relabel:

• Bazel Build & Proofs pulls the Rocq toolchain through Nix. Rivet Validation proves rust-cpu boxes can cargo install from git, which is not the same as having a working Nix daemon. Verify before moving, or it becomes a required check that fails environmentally — which reads exactly like a miscompile and is the worst possible failure mode for this gate.
• Z3 Verification is the feature-gated differential oracle and needs Z3 headers present.

The ordering constraint from the earlier comment stands and matters most here: a required context naming a label no online runner satisfies does not fail — it never runs, and that deadlocks every merge in the repo. So: move a non-required ubuntu-latest job to rust-cpu first and watch one full cycle; then move required jobs one at a time, each with its own green cycle before the next; and never move Bazel or Z3 on the same PR as anything else.

Measured while four v0.60 PRs sat at 3/9 with 8 runners idle.