Skip to content

fix(ci): bound the jury queue — concurrency group + verdict off the pool - #5

Open
todie wants to merge 1 commit into
mainfrom
fix/jury-concurrency-and-verdict-runner
Open

fix(ci): bound the jury queue — concurrency group + verdict off the pool#5
todie wants to merge 1 commit into
mainfrom
fix/jury-concurrency-and-verdict-runner

Conversation

@todie

@todie todie commented Aug 16, 2026

Copy link
Copy Markdown
Contributor

The jury was not failing. It was starving, and the queue was unbounded.

Found while auditing why two conversion PRs had sat with a queued jury for six hours.

Measured

repo queued jury runs branches oldest
cerebral-work/terrarium 6 5 10:00
cerebral-work/linearctl 7 5 yesterday 21:07 (~20h)

Most belonged to other people's PRs, not to the conversion work — docs/celestial-mechanics-prospects, docs/canon-rollout-stage, fix/release-sync-funnel-containment, feat/queue-hmac-verify, docs/actuator-contracts. An idle runner was registered the entire time.

Two structural causes, both in this file

1. No concurrency group

Nothing cancels a superseded run by default. Every push to a PR left the previous run queued forever. One branch pushed three times contributed three permanently-queued runs; on a scale-to-zero ARC pool that accumulates until it starves every repo pointed at that pool.

concurrency:
  group: agent-jury-${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

github.ref is per-PR for pull_request, so this only supersedes older runs of the same pull request.

2. The verdict job burned a second self-hosted slot to run echo

verdict reads one output from needs and exits. No network, no secrets, no cluster access — yet it inherited inputs.runner. Every review cost two slots on pools where the slot is the scarce resource (cerebral-work's reverie pool is max 3 and was running one).

New verdict_runner input, default ubuntu-latest. A GitHub-hosted minute is a better trade than a slot in a starved pool. Overridable for repos that cannot use GitHub-hosted runners.

Together these halve per-review demand and stop the backlog from growing.

What this does not fix

The underlying capacity problem is separate and still open: the runner fleets reserve 2–6 CPU each while idling at ~1 millicore, so the pool cannot actually scale to its max. That is unsigned-action-runners#33.

Housekeeping done alongside

Cancelled only my own superseded runs (3 across the two repos). The other queued runs belong to other people's PRs and are not mine to cancel — but they are starved by the same cause and should drain once this lands and capacity is fixed.

The jury was not failing. It was starving, and the queue was unbounded.

MEASURED 2026-08-16 on cerebral-work: terrarium had 6 and linearctl 7 jury runs
sitting QUEUED, spanning FIVE branches each. Most belonged to other people's
PRs, not to the conversion work. The oldest had been queued roughly 20 hours.
An idle runner was registered the whole time.

TWO CAUSES, both structural, both in this file.

1. No concurrency group. Every push to a PR left the previous run queued
   forever — nothing cancels a superseded run by default. One branch pushed
   three times contributed three permanently-queued runs. On a scale-to-zero
   ARC pool that accumulates until it starves every repo pointed at that pool.
   Fixed with a per-ref concurrency group and cancel-in-progress. github.ref is
   per-PR for pull_request, so this only supersedes older runs of the SAME pull
   request.

2. The verdict job consumed a second self-hosted slot to run `echo` and `exit`.
   It reads one output from `needs`; it has no network, no secrets and no
   cluster access, yet it inherited `inputs.runner`. Every review therefore
   cost TWO slots on pools where the slot is the scarce resource —
   cerebral-work's reverie pool is max 3 and was running one. New
   `verdict_runner` input, default ubuntu-latest; a GitHub-hosted minute is a
   better trade than a slot in a starved pool. Overridable for repos that
   cannot use GitHub-hosted runners.

Together these halve per-review demand and stop the backlog from growing.

Note this does NOT fix the underlying capacity problem, which is separate: the
runner fleets reserve 2-6 CPU each while sitting at ~1 millicore idle, so the
pool cannot actually scale to its max. That is unsigned-action-runners#33.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant