fix(ci): bound the jury queue — concurrency group + verdict off the pool - #5
Open
todie wants to merge 1 commit into
Open
fix(ci): bound the jury queue — concurrency group + verdict off the pool#5todie wants to merge 1 commit into
todie wants to merge 1 commit into
Conversation
The jury was not failing. It was starving, and the queue was unbounded. MEASURED 2026-08-16 on cerebral-work: terrarium had 6 and linearctl 7 jury runs sitting QUEUED, spanning FIVE branches each. Most belonged to other people's PRs, not to the conversion work. The oldest had been queued roughly 20 hours. An idle runner was registered the whole time. TWO CAUSES, both structural, both in this file. 1. No concurrency group. Every push to a PR left the previous run queued forever — nothing cancels a superseded run by default. One branch pushed three times contributed three permanently-queued runs. On a scale-to-zero ARC pool that accumulates until it starves every repo pointed at that pool. Fixed with a per-ref concurrency group and cancel-in-progress. github.ref is per-PR for pull_request, so this only supersedes older runs of the SAME pull request. 2. The verdict job consumed a second self-hosted slot to run `echo` and `exit`. It reads one output from `needs`; it has no network, no secrets and no cluster access, yet it inherited `inputs.runner`. Every review therefore cost TWO slots on pools where the slot is the scarce resource — cerebral-work's reverie pool is max 3 and was running one. New `verdict_runner` input, default ubuntu-latest; a GitHub-hosted minute is a better trade than a slot in a starved pool. Overridable for repos that cannot use GitHub-hosted runners. Together these halve per-review demand and stop the backlog from growing. Note this does NOT fix the underlying capacity problem, which is separate: the runner fleets reserve 2-6 CPU each while sitting at ~1 millicore idle, so the pool cannot actually scale to its max. That is unsigned-action-runners#33.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The jury was not failing. It was starving, and the queue was unbounded.
Found while auditing why two conversion PRs had sat with a queued jury for six hours.
Measured
Most belonged to other people's PRs, not to the conversion work —
docs/celestial-mechanics-prospects,docs/canon-rollout-stage,fix/release-sync-funnel-containment,feat/queue-hmac-verify,docs/actuator-contracts. An idle runner was registered the entire time.Two structural causes, both in this file
1. No concurrency group
Nothing cancels a superseded run by default. Every push to a PR left the previous run queued forever. One branch pushed three times contributed three permanently-queued runs; on a scale-to-zero ARC pool that accumulates until it starves every repo pointed at that pool.
github.refis per-PR forpull_request, so this only supersedes older runs of the same pull request.2. The verdict job burned a second self-hosted slot to run
echoverdictreads one output fromneedsand exits. No network, no secrets, no cluster access — yet it inheritedinputs.runner. Every review cost two slots on pools where the slot is the scarce resource (cerebral-work'sreveriepool is max 3 and was running one).New
verdict_runnerinput, defaultubuntu-latest. A GitHub-hosted minute is a better trade than a slot in a starved pool. Overridable for repos that cannot use GitHub-hosted runners.Together these halve per-review demand and stop the backlog from growing.
What this does not fix
The underlying capacity problem is separate and still open: the runner fleets reserve 2–6 CPU each while idling at ~1 millicore, so the pool cannot actually scale to its max. That is
unsigned-action-runners#33.Housekeeping done alongside
Cancelled only my own superseded runs (3 across the two repos). The other queued runs belong to other people's PRs and are not mine to cancel — but they are starved by the same cause and should drain once this lands and capacity is fixed.