From 79eb29aa66619fd9c7c73b05b58f60f00b5f1b69 Mon Sep 17 00:00:00 2001 From: Buck Doyle Date: Wed, 19 Aug 2026 13:49:05 -0400 Subject: [PATCH 01/15] Sample the GitHub Actions queue into Loki and chart it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit GitHub's Actions UI cannot say how deep the queue is, how long jobs have been waiting, or which branches and people are holding the runners. Establishing that during the recent archive outage meant scripting the REST API by hand, and the answer disappeared as soon as the terminal scrolled. A collector samples the Actions API and prints one JSON line per observation, in the same shape every service here already logs, so the existing Alloy and FireLens pipelines carry it to Loki without special handling. A dashboard in the boxel-status folder reads it back: queue depth and wait percentiles over time, runners consumed by branch, author and workflow, and a table of in-progress jobs ranked by how long their current step has run — the signal that separates a wedged job from a slow one. Grouped depth is emitted as its own lines rather than nested on the snapshot, because LogQL flattens nested JSON into one label per key, which for branch names yields a label per branch instead of a series grouped by branch. Two constraints shape the collector. It must not run as a scheduled Actions workflow, since it would queue behind the backlog it measures and go blind during the incident it exists for. And a sample costs one request per active run, so at the ~60 runs this repository sustains, a one-minute interval would consume most of a token's hourly REST budget; the interval is two minutes and a sample is skipped rather than reported partially when the remaining budget nears its reserve. Co-Authored-By: Claude Opus 5 --- packages/observability/README.md | 47 ++ .../observability/collectors/actions-queue.ts | 371 +++++++++ .../boxel-status/actions-queue.json | 769 ++++++++++++++++++ 3 files changed, 1187 insertions(+) create mode 100644 packages/observability/collectors/actions-queue.ts create mode 100644 packages/observability/grafanactl/resources/dashboards/boxel-status/actions-queue.json diff --git a/packages/observability/README.md b/packages/observability/README.md index af3ca6da876..3fc74411635 100644 --- a/packages/observability/README.md +++ b/packages/observability/README.md @@ -41,6 +41,10 @@ provisioning/ # mounted into Grafana at /etc/grafana/provisioning/ alerting/ # alert rule groups, contact points, notification policies local-only/ # local-dev overrides — bind-mounted file-by-file over # `datasources/`. apply-datasources.sh ignores this dir. +collectors/ + actions-queue.ts # samples the GitHub Actions REST API and prints one JSON + # line per observation on channel `boxel:actions-queue`; + # feeds the "GitHub Actions Queue" dashboard alloy/ config.alloy # local log scraper config — discovers Docker containers # and ships their stdout into Loki @@ -69,6 +73,49 @@ docker-compose.yml # local Grafana 12.4.3 + Loki 3.4.4 + Alloy 1.10.0 # + Prometheus 3.0.0 (scrapes synapse) ``` +## Collectors + +Some signals have no service to emit them, so a collector samples an external +API and prints the result as JSON log lines on stdout — the same shape every +other service logs in, so the existing Alloy (local) and FireLens (hosted) +pipelines carry it to Loki with no special handling. + +### actions-queue + +Samples GitHub Actions queue depth, per-job wait times and runner consumption. + +```bash +GITHUB_TOKEN=$(gh auth token) node collectors/actions-queue.ts --once +GITHUB_TOKEN=$(gh auth token) node collectors/actions-queue.ts # loops, default 120s +``` + +It emits four event types, all on channel `boxel:actions-queue`: + +| `event_type` | one line per | carries | +| ---------------------------------------- | ------------------------- | ------------------------------------------------------------------------------------------- | +| `job` | queued or running job | branch, actor, workflow, runner labels, `queued_seconds`, `running_seconds`, `current_step` | +| `group` | dimension × key | `dimension` (branch / actor / workflow), `key`, `queued`, `running` | +| `snapshot` | sample | `queued_jobs`, `running_jobs`, `active_runs`, `rate_limit_remaining` | +| `collector-error`, `collector-throttled` | failure or skipped sample | why the series has a gap | + +Grouped depth is emitted as its own lines rather than nested on the snapshot +because LogQL flattens nested JSON into one label per key, which for dynamic +keys like branch names produces a label per branch instead of a series that can +be grouped by branch. + +Two constraints are load-bearing: + +- **It must not run as a scheduled GitHub Actions workflow.** It would queue + behind the backlog it measures and go blind during the incident it exists + for. Hosted, it belongs on a schedule outside Actions. +- **A sample costs one request per active run, plus two.** At ~60 runs in + flight a 60-second interval would spend most of a token's 5,000 hourly REST + requests, so the default interval is 120s and sampling stops rather than + reporting partial depth when the remaining budget nears its reserve. + +The dashboard reads `{service="actions-collector", env="$env"}`, so a hosted +deployment needs to log under that service name. + ## Local workflow ```sh diff --git a/packages/observability/collectors/actions-queue.ts b/packages/observability/collectors/actions-queue.ts new file mode 100644 index 00000000000..e48980286a8 --- /dev/null +++ b/packages/observability/collectors/actions-queue.ts @@ -0,0 +1,371 @@ +// Samples the GitHub Actions queue and emits one JSON log line per observation, +// which Alloy (local) or FireLens (hosted) ships to Loki for the "GitHub Actions +// Queue" dashboard to read with `| json`. +// +// Why a poller rather than webhooks: the questions this answers are about +// *standing* state — how deep is the queue right now, how long has this job been +// waiting, which branches are holding runners. Webhooks deliver transitions, so +// reconstructing depth from them means keeping state and healing missed +// deliveries. A periodic snapshot is inherently self-correcting. +// +// This must not run as a scheduled GitHub Actions workflow: it would queue +// behind the very backlog it measures and go blind during the incident it +// exists for. + +const CHANNEL = 'boxel:actions-queue'; + +// One sample costs two run-list requests plus one per active run. A busy hour +// on this repository has ~60 runs in flight, so a 60-second interval would spend +// ~3,800 of the 5,000 hourly REST requests a token is allowed — enough to +// starve anything else using the same token. Two minutes halves that, and the +// reserve below stops a spike from consuming the rest. +const DEFAULT_INTERVAL_SECONDS = 120; + +// Requests deliberately left unspent, so a burst of runs can never take the +// token to zero and lock out other consumers. +const RATE_LIMIT_RESERVE = 750; + +interface Job { + id: number; + run_id: number; + run_attempt: number; + workflow_name: string | null; + head_branch: string | null; + name: string; + status: string; + conclusion: string | null; + created_at: string; + started_at: string | null; + runner_name: string | null; + labels: string[]; + steps?: { name: string; status: string; started_at: string | null }[]; +} + +interface Run { + id: number; + name: string | null; + head_branch: string | null; + run_attempt: number; + created_at: string; + status: string; + actor?: { login?: string } | null; + triggering_actor?: { login?: string } | null; + event?: string; +} + +interface Options { + repo: string; + token: string; + intervalMs: number; + once: boolean; +} + +function usage(): never { + console.error( + `usage: node actions-queue.ts [--repo owner/name] [--interval seconds] [--once] + + --repo Repository to sample. Default $GITHUB_REPOSITORY, else cardstack/boxel. + --interval Seconds between samples. Default ${DEFAULT_INTERVAL_SECONDS}. + --once Take a single sample and exit, rather than looping. + + Requires GITHUB_TOKEN with \`actions: read\` on the repository.`, + ); + process.exit(2); +} + +function parseArgs(argv: string[]): Options { + let repo = process.env.GITHUB_REPOSITORY || 'cardstack/boxel'; + let intervalSeconds = Number( + process.env.ACTIONS_QUEUE_INTERVAL || DEFAULT_INTERVAL_SECONDS, + ); + let once = false; + + for (let i = 0; i < argv.length; i++) { + const arg = argv[i]; + if (arg === '--once') { + once = true; + } else if (arg === '--repo') { + repo = argv[++i] ?? usage(); + } else if (arg === '--interval') { + intervalSeconds = Number(argv[++i]); + } else { + usage(); + } + } + + const token = process.env.GITHUB_TOKEN || ''; + if (!token) { + console.error('GITHUB_TOKEN is not set'); + process.exit(2); + } + if (!/^[^/]+\/[^/]+$/.test(repo)) { + console.error(`--repo must be owner/name, got "${repo}"`); + process.exit(2); + } + if (!Number.isFinite(intervalSeconds) || intervalSeconds < 10) { + console.error('--interval must be a number of seconds, at least 10'); + process.exit(2); + } + return { repo, token, intervalMs: intervalSeconds * 1000, once }; +} + +// Emitted on its own line so a log-shipping pipeline sees one JSON object per +// line. Anything non-serialisable would corrupt the stream, so values are kept +// to primitives and arrays of primitives by construction. +function emit(record: Record): void { + process.stdout.write(JSON.stringify({ channel: CHANNEL, ...record }) + '\n'); +} + +let rateLimitRemaining: number | null = null; + +async function gh(url: string, token: string): Promise { + const res = await fetch(url, { + headers: { + authorization: `Bearer ${token}`, + accept: 'application/vnd.github+json', + 'x-github-api-version': '2022-11-28', + 'user-agent': 'boxel-actions-queue-collector', + }, + }); + const remaining = res.headers.get('x-ratelimit-remaining'); + if (remaining !== null) { + rateLimitRemaining = Number(remaining); + } + if (!res.ok) { + throw new Error(`GET ${url} → ${res.status} ${res.statusText}`); + } + return (await res.json()) as T; +} + +// The REST list endpoints cap at 100 per page. Callers here are bounded by how +// much work is in flight, not by repository history, so a small page cap is a +// backstop against a runaway loop rather than a real limit. +async function paginate( + url: string, + token: string, + extract: (page: any) => T[], + maxPages = 10, +): Promise { + const items: T[] = []; + for (let page = 1; page <= maxPages; page++) { + const body = await gh( + `${url}${url.includes('?') ? '&' : '?'}per_page=100&page=${page}`, + token, + ); + const batch = extract(body); + items.push(...batch); + if (batch.length < 100) break; + } + return items; +} + +function seconds(from: string | null | undefined, to: number): number | null { + if (!from) return null; + const t = Date.parse(from); + return Number.isFinite(t) ? Math.max(0, Math.round((to - t) / 1000)) : null; +} + +async function sample(opts: Options): Promise { + const base = `https://api.github.com/repos/${opts.repo}/actions`; + const observedAt = Date.now(); + const observed_at = new Date(observedAt).toISOString(); + + // Only queued and in_progress runs can hold or want a runner, so completed + // history is deliberately not fetched — it would dominate the request budget + // without informing the queue. + const runs: Run[] = []; + for (const status of ['queued', 'in_progress']) { + runs.push( + ...(await paginate( + `${base}/runs?status=${status}`, + opts.token, + (b) => b.workflow_runs ?? [], + )), + ); + } + + const runById = new Map(runs.map((r) => [r.id, r])); + + // Fetching one job list per run is the expensive half of a sample. Skipping + // the whole sample keeps the series honest: a partial sample would understate + // queue depth, which is worse than a visible gap. + if ( + rateLimitRemaining !== null && + rateLimitRemaining - runs.length < RATE_LIMIT_RESERVE + ) { + emit({ + event_type: 'collector-throttled', + observed_at, + repo: opts.repo, + active_runs: runs.length, + rate_limit_remaining: rateLimitRemaining, + }); + return; + } + + let queuedJobs = 0; + let runningJobs = 0; + // Grouped depth is emitted as its own lines rather than as nested objects on + // the snapshot, because LogQL flattens nested JSON into one label per key — + // which for dynamic keys like branch names yields an unqueryable label per + // branch instead of a series grouped by branch. + const groups: Record< + string, + Map + > = { + workflow: new Map(), + branch: new Map(), + actor: new Map(), + }; + + for (const run of runs) { + let jobs: Job[]; + try { + jobs = await paginate( + `${base}/runs/${run.id}/jobs`, + opts.token, + (b) => b.jobs ?? [], + ); + } catch (e) { + // A run can complete and be reaped between listing and this fetch. That is + // ordinary, so it must not abort the whole sample. + emit({ + event_type: 'collector-error', + observed_at, + repo: opts.repo, + run_id: run.id, + message: e instanceof Error ? e.message : String(e), + }); + continue; + } + + for (const job of jobs) { + if (job.status !== 'queued' && job.status !== 'in_progress') continue; + + const meta = runById.get(job.run_id); + const actor = meta?.triggering_actor?.login ?? meta?.actor?.login ?? null; + const branch = job.head_branch ?? meta?.head_branch ?? null; + const workflow = job.workflow_name ?? meta?.name ?? null; + + // While a job is queued GitHub reports started_at equal to created_at, so + // it cannot be used to tell waiting from running. Queue wait is therefore + // measured from created_at, and only a job that has actually left the + // queue reports a running duration. + const queued_seconds = + job.status === 'queued' + ? seconds(job.created_at, observedAt) + : seconds( + job.created_at, + Date.parse(job.started_at ?? job.created_at), + ); + const running_seconds = + job.status === 'in_progress' + ? seconds(job.started_at, observedAt) + : null; + + const step = job.steps?.find((s) => s.status === 'in_progress'); + + if (job.status === 'queued') queuedJobs++; + else runningJobs++; + + const bump = (dimension: string, key: string | null) => { + if (!key) return; + const m = groups[dimension]; + const entry = m.get(key) ?? { queued: 0, running: 0 }; + if (job.status === 'queued') entry.queued++; + else entry.running++; + m.set(key, entry); + }; + bump('workflow', workflow); + bump('branch', branch); + bump('actor', actor); + + emit({ + event_type: 'job', + observed_at, + repo: opts.repo, + workflow, + run_id: job.run_id, + run_attempt: job.run_attempt, + job_id: job.id, + job: job.name, + status: job.status, + head_branch: branch, + actor, + event: meta?.event ?? null, + runner_name: job.runner_name || null, + runner_labels: job.labels ?? [], + created_at: job.created_at, + started_at: job.started_at, + queued_seconds, + running_seconds, + current_step: step?.name ?? null, + current_step_seconds: seconds(step?.started_at ?? null, observedAt), + }); + } + } + + for (const [dimension, m] of Object.entries(groups)) { + for (const [key, counts] of m) { + emit({ + event_type: 'group', + observed_at, + repo: opts.repo, + dimension, + key, + queued: counts.queued, + running: counts.running, + total: counts.queued + counts.running, + }); + } + } + + // A standalone total, so queue depth is one field rather than a count over + // the per-job lines — the per-job series is subject to Loki's retention and + // to sampling gaps, and depth should survive both. + emit({ + event_type: 'snapshot', + observed_at, + repo: opts.repo, + active_runs: runs.length, + queued_jobs: queuedJobs, + running_jobs: runningJobs, + total_jobs: queuedJobs + runningJobs, + rate_limit_remaining: rateLimitRemaining, + }); +} + +async function main(): Promise { + const opts = parseArgs(process.argv.slice(2)); + let stopping = false; + for (const sig of ['SIGINT', 'SIGTERM'] as const) { + process.on(sig, () => { + stopping = true; + }); + } + + for (;;) { + const startedAt = Date.now(); + try { + await sample(opts); + } catch (e) { + // One failed sample should not end a long-running collector; the next tick + // re-reads the whole state anyway. + emit({ + event_type: 'collector-error', + observed_at: new Date().toISOString(), + repo: opts.repo, + message: e instanceof Error ? e.message : String(e), + }); + if (opts.once) process.exitCode = 1; + } + if (opts.once || stopping) return; + const elapsed = Date.now() - startedAt; + await new Promise((r) => + setTimeout(r, Math.max(0, opts.intervalMs - elapsed)), + ); + if (stopping) return; + } +} + +await main(); diff --git a/packages/observability/grafanactl/resources/dashboards/boxel-status/actions-queue.json b/packages/observability/grafanactl/resources/dashboards/boxel-status/actions-queue.json new file mode 100644 index 00000000000..298a7eae2fa --- /dev/null +++ b/packages/observability/grafanactl/resources/dashboards/boxel-status/actions-queue.json @@ -0,0 +1,769 @@ +{ + "apiVersion": "dashboard.grafana.app/v1beta1", + "kind": "Dashboard", + "metadata": { + "annotations": { + "grafana.app/folder": "defd2d156sav4d" + }, + "name": "boxel-actions-queue" + }, + "spec": { + "annotations": { + "list": [] + }, + "description": "GitHub Actions queue depth, wait times and runner consumption, sampled from the Actions REST API and shipped as JSON log lines on channel boxel:actions-queue.", + "editable": true, + "fiscalYearStartMonth": 0, + "graphTooltip": 0, + "links": [], + "panels": [ + { + "id": 1, + "type": "text", + "title": "About this dashboard", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "x": 0, + "y": 0, + "w": 24, + "h": 4 + }, + "targets": [], + "fieldConfig": { + "defaults": { + "custom": {} + }, + "overrides": [] + }, + "options": { + "mode": "markdown", + "content": "Sampled from the GitHub Actions REST API by `packages/observability/collectors/actions-queue.ts`, which emits one JSON line per observation on channel `boxel:actions-queue`.\n\nThe collector runs **outside** GitHub Actions on purpose — a scheduled workflow would queue behind the backlog it measures and go blind during the incident it exists for.\n\nDepth and grouped depth come from periodic snapshots, so each point is a sample rather than a continuous measure; gaps mean the collector was down or throttled (see *Collector health*)." + } + }, + { + "id": 2, + "type": "row", + "title": "Queue depth", + "collapsed": false, + "gridPos": { + "x": 0, + "y": 4, + "w": 24, + "h": 1 + }, + "panels": [] + }, + { + "id": 3, + "type": "stat", + "title": "Jobs queued now", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "x": 0, + "y": 5, + "w": 4, + "h": 4 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "editorMode": "code", + "expr": "last_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"snapshot\" | unwrap queued_jobs [$__interval])", + "refId": "A", + "queryType": "range", + "legendFormat": "queued" + } + ], + "fieldConfig": { + "defaults": { + "custom": {} + }, + "overrides": [] + }, + "options": { + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "auto", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "textMode": "auto", + "wideLayout": true + }, + "description": "Jobs waiting for a runner at the most recent sample." + }, + { + "id": 4, + "type": "stat", + "title": "Jobs running now", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "x": 4, + "y": 5, + "w": 4, + "h": 4 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "editorMode": "code", + "expr": "last_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"snapshot\" | unwrap running_jobs [$__interval])", + "refId": "A", + "queryType": "range", + "legendFormat": "running" + } + ], + "fieldConfig": { + "defaults": { + "custom": {} + }, + "overrides": [] + }, + "options": { + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "auto", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "textMode": "auto", + "wideLayout": true + }, + "description": "Jobs currently holding a runner." + }, + { + "id": 5, + "type": "stat", + "title": "Active runs", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "x": 8, + "y": 5, + "w": 4, + "h": 4 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "editorMode": "code", + "expr": "last_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"snapshot\" | unwrap active_runs [$__interval])", + "refId": "A", + "queryType": "range", + "legendFormat": "runs" + } + ], + "fieldConfig": { + "defaults": { + "custom": {} + }, + "overrides": [] + }, + "options": { + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "auto", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "textMode": "auto", + "wideLayout": true + }, + "description": "Workflow runs in queued or in_progress state." + }, + { + "id": 6, + "type": "timeseries", + "title": "Queue depth over time", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "x": 12, + "y": 5, + "w": 12, + "h": 8 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "editorMode": "code", + "expr": "max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"snapshot\" | unwrap queued_jobs [$__interval])", + "refId": "A", + "queryType": "range", + "legendFormat": "queued" + }, + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "editorMode": "code", + "expr": "max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"snapshot\" | unwrap running_jobs [$__interval])", + "refId": "B", + "queryType": "range", + "legendFormat": "running" + } + ], + "fieldConfig": { + "defaults": { + "custom": {} + }, + "overrides": [] + }, + "options": {}, + "description": "Queued depth rising while running stays flat means the pool is saturated, not slow." + }, + { + "id": 7, + "type": "stat", + "title": "Longest current wait", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "x": 0, + "y": 9, + "w": 12, + "h": 4 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "editorMode": "code", + "expr": "max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"job\" | status=\"queued\" | unwrap queued_seconds [$__interval])", + "refId": "A", + "queryType": "range", + "legendFormat": "longest wait" + } + ], + "fieldConfig": { + "defaults": { + "custom": {}, + "unit": "s" + }, + "overrides": [] + }, + "options": { + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "auto", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "textMode": "auto", + "wideLayout": true + }, + "description": "Age of the oldest job still waiting for a runner." + }, + { + "id": 8, + "type": "row", + "title": "Wait times", + "collapsed": false, + "gridPos": { + "x": 0, + "y": 13, + "w": 24, + "h": 1 + }, + "panels": [] + }, + { + "id": 9, + "type": "timeseries", + "title": "Queue wait percentiles", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "x": 0, + "y": 14, + "w": 24, + "h": 8 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "editorMode": "code", + "expr": "quantile_over_time(0.5, {service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"job\" | status=\"queued\" | unwrap queued_seconds [$__interval])", + "refId": "A", + "queryType": "range", + "legendFormat": "p50" + }, + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "editorMode": "code", + "expr": "quantile_over_time(0.9, {service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"job\" | status=\"queued\" | unwrap queued_seconds [$__interval])", + "refId": "B", + "queryType": "range", + "legendFormat": "p90" + }, + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "editorMode": "code", + "expr": "max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"job\" | status=\"queued\" | unwrap queued_seconds [$__interval])", + "refId": "C", + "queryType": "range", + "legendFormat": "max" + } + ], + "fieldConfig": { + "defaults": { + "custom": {}, + "unit": "s" + }, + "overrides": [] + }, + "options": {}, + "description": "How long jobs have been waiting, measured from job creation." + }, + { + "id": 10, + "type": "row", + "title": "Who is consuming the pool", + "collapsed": false, + "gridPos": { + "x": 0, + "y": 22, + "w": 24, + "h": 1 + }, + "panels": [] + }, + { + "id": 11, + "type": "timeseries", + "title": "Queued jobs by branch", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "x": 0, + "y": 23, + "w": 8, + "h": 8 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "editorMode": "code", + "expr": "sum by (key) (max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"group\" | dimension=\"branch\" | unwrap queued [$__interval]))", + "refId": "A", + "queryType": "range", + "legendFormat": "{{key}}" + } + ], + "fieldConfig": { + "defaults": { + "custom": {} + }, + "overrides": [] + }, + "options": {}, + "description": "Queued jobs grouped by branch, from the collector's per-group snapshot lines." + }, + { + "id": 12, + "type": "timeseries", + "title": "Queued jobs by author", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "x": 8, + "y": 23, + "w": 8, + "h": 8 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "editorMode": "code", + "expr": "sum by (key) (max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"group\" | dimension=\"actor\" | unwrap queued [$__interval]))", + "refId": "A", + "queryType": "range", + "legendFormat": "{{key}}" + } + ], + "fieldConfig": { + "defaults": { + "custom": {} + }, + "overrides": [] + }, + "options": {}, + "description": "Queued jobs grouped by actor, from the collector's per-group snapshot lines." + }, + { + "id": 13, + "type": "timeseries", + "title": "Queued jobs by workflow", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "x": 16, + "y": 23, + "w": 8, + "h": 8 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "editorMode": "code", + "expr": "sum by (key) (max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"group\" | dimension=\"workflow\" | unwrap queued [$__interval]))", + "refId": "A", + "queryType": "range", + "legendFormat": "{{key}}" + } + ], + "fieldConfig": { + "defaults": { + "custom": {} + }, + "overrides": [] + }, + "options": {}, + "description": "Queued jobs grouped by workflow, from the collector's per-group snapshot lines." + }, + { + "id": 14, + "type": "timeseries", + "title": "Running jobs by branch", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "x": 0, + "y": 31, + "w": 8, + "h": 8 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "editorMode": "code", + "expr": "sum by (key) (max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"group\" | dimension=\"branch\" | unwrap running [$__interval]))", + "refId": "A", + "queryType": "range", + "legendFormat": "{{key}}" + } + ], + "fieldConfig": { + "defaults": { + "custom": {} + }, + "overrides": [] + }, + "options": {}, + "description": "Runners currently held, grouped by branch." + }, + { + "id": 15, + "type": "timeseries", + "title": "Running jobs by author", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "x": 8, + "y": 31, + "w": 8, + "h": 8 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "editorMode": "code", + "expr": "sum by (key) (max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"group\" | dimension=\"actor\" | unwrap running [$__interval]))", + "refId": "A", + "queryType": "range", + "legendFormat": "{{key}}" + } + ], + "fieldConfig": { + "defaults": { + "custom": {} + }, + "overrides": [] + }, + "options": {}, + "description": "Runners currently held, grouped by actor." + }, + { + "id": 16, + "type": "timeseries", + "title": "Running jobs by workflow", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "x": 16, + "y": 31, + "w": 8, + "h": 8 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "editorMode": "code", + "expr": "sum by (key) (max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"group\" | dimension=\"workflow\" | unwrap running [$__interval]))", + "refId": "A", + "queryType": "range", + "legendFormat": "{{key}}" + } + ], + "fieldConfig": { + "defaults": { + "custom": {} + }, + "overrides": [] + }, + "options": {}, + "description": "Runners currently held, grouped by workflow." + }, + { + "id": 17, + "type": "row", + "title": "Long-running steps", + "collapsed": false, + "gridPos": { + "x": 0, + "y": 39, + "w": 24, + "h": 1 + }, + "panels": [] + }, + { + "id": 18, + "type": "table", + "title": "In-progress jobs by current step duration", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "x": 0, + "y": 40, + "w": 24, + "h": 10 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "editorMode": "code", + "expr": "topk(25, max by (job, head_branch, workflow, current_step) (last_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"job\" | status=\"in_progress\" | unwrap current_step_seconds [10m])))", + "refId": "A", + "queryType": "instant" + } + ], + "fieldConfig": { + "defaults": { + "custom": {}, + "unit": "s" + }, + "overrides": [] + }, + "options": {}, + "description": "A job whose current step has run far longer than its peers is the wedge signal — compare against the same step on other shards before concluding it is stuck." + }, + { + "id": 19, + "type": "row", + "title": "Collector health", + "collapsed": false, + "gridPos": { + "x": 0, + "y": 50, + "w": 24, + "h": 1 + }, + "panels": [] + }, + { + "id": 20, + "type": "timeseries", + "title": "GitHub API requests remaining", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "x": 0, + "y": 51, + "w": 12, + "h": 6 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "editorMode": "code", + "expr": "min_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"snapshot\" | unwrap rate_limit_remaining [$__interval])", + "refId": "A", + "queryType": "range", + "legendFormat": "remaining" + } + ], + "fieldConfig": { + "defaults": { + "custom": {} + }, + "overrides": [] + }, + "options": {}, + "description": "Sampling stops rather than reporting partial depth when this nears the reserve." + }, + { + "id": 21, + "type": "logs", + "title": "Collector errors and throttling", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "x": 12, + "y": 51, + "w": 12, + "h": 6 + }, + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "editorMode": "code", + "expr": "{service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=~\"collector-error|collector-throttled\"", + "refId": "A", + "queryType": "range" + } + ], + "fieldConfig": { + "defaults": { + "custom": {} + }, + "overrides": [] + }, + "options": {}, + "description": "Gaps in the series above should have a matching line here." + } + ], + "refresh": "1m", + "schemaVersion": 42, + "tags": [ + "ci", + "actions", + "forensics" + ], + "templating": { + "list": [ + { + "hide": 2, + "name": "env", + "query": "__ENV__", + "skipUrlSync": true, + "type": "constant" + } + ] + }, + "time": { + "from": "now-6h", + "to": "now" + }, + "timepicker": {}, + "timezone": "browser", + "title": "GitHub Actions Queue", + "weekStart": "" + } +} From de79117f9dd343b2833de37e06fdda977a478e29 Mon Sep 17 00:00:00 2001 From: Buck Doyle Date: Wed, 19 Aug 2026 14:14:57 -0400 Subject: [PATCH 02/15] Package the Actions queue collector so it can run in staging and production MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The collector is a single file that needs nothing beyond the Node runtime and global fetch, so its image is the runtime plus that file — no package install and no lockfile. Deployment is workflow_dispatch only. Putting the deploy of a queue sampler on a CI trigger would make it queue behind the backlog it reports on, which is the same reason the collector itself does not run inside Actions. The hosted task reads its credential from ACTIONS_COLLECTOR_GITHUB_TOKEN rather than a bare GITHUB_TOKEN, because its SSM path is shared with every other boxel service and a generic name there would be ambiguous. The plain name still works for local runs. Co-Authored-By: Claude Opus 5 --- .../manual-deploy-actions-collector.yml | 44 +++++++++++++++++++ packages/observability/README.md | 17 +++++++ packages/observability/collectors/Dockerfile | 12 +++++ .../observability/collectors/actions-queue.ts | 13 +++++- 4 files changed, 84 insertions(+), 2 deletions(-) create mode 100644 .github/workflows/manual-deploy-actions-collector.yml create mode 100644 packages/observability/collectors/Dockerfile diff --git a/.github/workflows/manual-deploy-actions-collector.yml b/.github/workflows/manual-deploy-actions-collector.yml new file mode 100644 index 00000000000..f558d51b664 --- /dev/null +++ b/.github/workflows/manual-deploy-actions-collector.yml @@ -0,0 +1,44 @@ +name: Manual Deploy [actions-collector] + +# Builds and deploys the GitHub Actions queue collector +# (packages/observability/collectors/actions-queue.ts), whose output feeds the +# "GitHub Actions Queue" Grafana dashboard. +# +# Deliberately not triggered by CI: this samples the Actions queue, so running +# its own deploy through a busy queue is the situation it exists to measure. + +on: + workflow_dispatch: + inputs: + environment: + description: Deployment environment + required: false + default: staging + +permissions: + contents: read + deployments: write + id-token: write + +jobs: + build-actions-collector: + name: Build actions-collector Docker image + uses: cardstack/gh-actions/.github/workflows/docker-ecr.yml@main + secrets: inherit + with: + repository: "boxel-actions-collector-${{ inputs.environment }}" + environment: ${{ inputs.environment }} + dockerfile: "packages/observability/collectors/Dockerfile" + + deploy-actions-collector: + needs: [build-actions-collector] + name: Deploy actions-collector to AWS ECS + uses: cardstack/gh-actions/.github/workflows/ecs-deploy.yml@main + secrets: inherit + with: + container-name: "boxel-actions-collector" + environment: ${{ inputs.environment }} + cluster: ${{ inputs.environment }} + service-name: "boxel-actions-collector-${{ inputs.environment }}" + image: ${{ needs.build-actions-collector.outputs.image }} + wait-for-service-stability: false diff --git a/packages/observability/README.md b/packages/observability/README.md index 3fc74411635..8b1c7377306 100644 --- a/packages/observability/README.md +++ b/packages/observability/README.md @@ -116,6 +116,23 @@ Two constraints are load-bearing: The dashboard reads `{service="actions-collector", env="$env"}`, so a hosted deployment needs to log under that service name. +#### Hosted + +Runs as a small ECS service — `cardstack/infra:configs/boxel-actions-collector` +— rather than a scheduled task: the signal is standing state sampled every +couple of minutes, and a per-invocation Fargate task would spend longer +starting than working. FireLens applies `service=actions-collector`, which is +what the dashboard selects on. + +Before the first apply, put a GitHub token with `actions: read` at +`//boxel/ACTIONS_COLLECTOR_GITHUB_TOKEN`. It is deliberately not managed +by Terraform, like the other secrets on that path. The collector accepts either +that name or a plain `GITHUB_TOKEN`, which is the convenient one locally. + +Build and deploy with the **Manual Deploy [actions-collector]** workflow. It is +`workflow_dispatch` only — putting the deploy of a queue sampler on a CI +trigger would make it queue behind the backlog it reports on. + ## Local workflow ```sh diff --git a/packages/observability/collectors/Dockerfile b/packages/observability/collectors/Dockerfile new file mode 100644 index 00000000000..65b1343d9d3 --- /dev/null +++ b/packages/observability/collectors/Dockerfile @@ -0,0 +1,12 @@ +ARG NODE_VERSION=24.17.0 +FROM node:${NODE_VERSION}-slim + +# The collector is a single file that uses nothing beyond the Node runtime and +# global fetch, so there is no package install step and no lockfile to copy — +# the image is the runtime plus one source file. +WORKDIR /app +COPY packages/observability/collectors/actions-queue.ts . + +# Runs in its default looping mode. Repository, sample interval and token come +# from the task environment; see packages/observability/README.md. +ENTRYPOINT ["node", "actions-queue.ts"] diff --git a/packages/observability/collectors/actions-queue.ts b/packages/observability/collectors/actions-queue.ts index e48980286a8..c6aabd1c972 100644 --- a/packages/observability/collectors/actions-queue.ts +++ b/packages/observability/collectors/actions-queue.ts @@ -93,9 +93,18 @@ function parseArgs(argv: string[]): Options { } } - const token = process.env.GITHUB_TOKEN || ''; + // The hosted task reads a dedicated parameter rather than a bare + // GITHUB_TOKEN, because its SSM path is shared with every other boxel + // service and a generic name there would be ambiguous. Locally the plain + // name is the convenient one, so both are accepted. + const token = + process.env.ACTIONS_COLLECTOR_GITHUB_TOKEN || + process.env.GITHUB_TOKEN || + ''; if (!token) { - console.error('GITHUB_TOKEN is not set'); + console.error( + 'set ACTIONS_COLLECTOR_GITHUB_TOKEN or GITHUB_TOKEN (needs actions: read)', + ); process.exit(2); } if (!/^[^/]+\/[^/]+$/.test(repo)) { From 149f436c3148202c4d9c4832458d8516c5cc75a9 Mon Sep 17 00:00:00 2001 From: Buck Doyle Date: Wed, 19 Aug 2026 14:26:11 -0400 Subject: [PATCH 03/15] Describe the collector's token by what it is actually for MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The token was documented as needing `actions: read`, which is wrong for this repository and misleading about why it exists. Both endpoints the collector reads answer unauthenticated on a public repository — the token's only job is the rate limit, which authentication raises from 60 requests an hour to 5,000, against a single sample that can cost sixty. So no permissions are needed while boxel is public. The private-repository requirement is stated separately, since that is when Actions: Read-only would start to matter. Co-Authored-By: Claude Opus 5 --- packages/observability/README.md | 17 +++++++++++++---- .../observability/collectors/actions-queue.ts | 7 +++++-- 2 files changed, 18 insertions(+), 6 deletions(-) diff --git a/packages/observability/README.md b/packages/observability/README.md index 8b1c7377306..e8b241e170f 100644 --- a/packages/observability/README.md +++ b/packages/observability/README.md @@ -124,10 +124,19 @@ couple of minutes, and a per-invocation Fargate task would spend longer starting than working. FireLens applies `service=actions-collector`, which is what the dashboard selects on. -Before the first apply, put a GitHub token with `actions: read` at -`//boxel/ACTIONS_COLLECTOR_GITHUB_TOKEN`. It is deliberately not managed -by Terraform, like the other secrets on that path. The collector accepts either -that name or a plain `GITHUB_TOKEN`, which is the convenient one locally. +Before the first apply, put a GitHub token at +`//boxel/ACTIONS_COLLECTOR_GITHUB_TOKEN` as a `SecureString`. It is +deliberately not managed by Terraform, like the other secrets on that path. The +collector accepts either that name or a plain `GITHUB_TOKEN`, which is the +convenient one locally. + +The token's only job is the rate limit: these endpoints are readable without +authentication on a public repository, but unauthenticated callers get 60 +requests an hour against the 5,000 an authenticated one gets, and a single +sample can cost sixty. So the token needs **no permissions** while `boxel` is +public — a fine-grained token scoped to _Public repositories (read-only)_, or +even a classic token with no scopes ticked, is enough. Were the repository ever +made private, it would need Actions: Read-only on it. Build and deploy with the **Manual Deploy [actions-collector]** workflow. It is `workflow_dispatch` only — putting the deploy of a queue sampler on a CI diff --git a/packages/observability/collectors/actions-queue.ts b/packages/observability/collectors/actions-queue.ts index c6aabd1c972..d290ab3b35a 100644 --- a/packages/observability/collectors/actions-queue.ts +++ b/packages/observability/collectors/actions-queue.ts @@ -68,7 +68,9 @@ function usage(): never { --interval Seconds between samples. Default ${DEFAULT_INTERVAL_SECONDS}. --once Take a single sample and exit, rather than looping. - Requires GITHUB_TOKEN with \`actions: read\` on the repository.`, + Requires a token only to raise the API rate limit from 60 to 5,000 requests + an hour. Against a public repository it needs no permissions at all; against + a private one it needs Actions: Read-only.`, ); process.exit(2); } @@ -103,7 +105,8 @@ function parseArgs(argv: string[]): Options { ''; if (!token) { console.error( - 'set ACTIONS_COLLECTOR_GITHUB_TOKEN or GITHUB_TOKEN (needs actions: read)', + 'set ACTIONS_COLLECTOR_GITHUB_TOKEN or GITHUB_TOKEN — any token raises the\n' + + 'rate limit from 60 to 5,000 requests an hour; see --help', ); process.exit(2); } From 15b9331558c8898d86aab73852581dd0a91b1821 Mon Sep 17 00:00:00 2001 From: Buck Doyle Date: Wed, 19 Aug 2026 16:41:50 -0400 Subject: [PATCH 04/15] Trigger the collector deploy from this branch, temporarily MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A workflow_dispatch workflow can only be dispatched once its file is on the default branch, so until this merges there is no way to build the collector — and with no collector running, the dashboard previews on this PR with every panel empty, which is indistinguishable from a query that is simply wrong. A push trigger scoped to this branch closes that loop, letting the queries be checked against real data before merging. The staging environment carries no branch policy and the OIDC trust policy for the deploy role admits any ref of this repository, so a branch build is permitted. `inputs.environment` is null on a push event, so the environment default is now explicit at each use — worth keeping after the trigger goes. Remove the push trigger before merge. Co-Authored-By: Claude Opus 5 --- .../manual-deploy-actions-collector.yml | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/.github/workflows/manual-deploy-actions-collector.yml b/.github/workflows/manual-deploy-actions-collector.yml index f558d51b664..d303bcc08ff 100644 --- a/.github/workflows/manual-deploy-actions-collector.yml +++ b/.github/workflows/manual-deploy-actions-collector.yml @@ -14,6 +14,13 @@ on: description: Deployment environment required: false default: staging + # TEMPORARY — remove before merge. A workflow_dispatch workflow can only be + # dispatched once its file is on the default branch, so until this merges + # there is no way to build the collector, and without a running collector the + # dashboard on this PR previews with every panel empty. This trigger closes + # that loop so the queries can be checked against real data before merging. + push: + branches: [cs-12571-make-the-github-actions-backlog-visible-in-grafana] permissions: contents: read @@ -26,8 +33,8 @@ jobs: uses: cardstack/gh-actions/.github/workflows/docker-ecr.yml@main secrets: inherit with: - repository: "boxel-actions-collector-${{ inputs.environment }}" - environment: ${{ inputs.environment }} + repository: "boxel-actions-collector-${{ inputs.environment || 'staging' }}" + environment: ${{ inputs.environment || 'staging' }} dockerfile: "packages/observability/collectors/Dockerfile" deploy-actions-collector: @@ -37,8 +44,8 @@ jobs: secrets: inherit with: container-name: "boxel-actions-collector" - environment: ${{ inputs.environment }} - cluster: ${{ inputs.environment }} - service-name: "boxel-actions-collector-${{ inputs.environment }}" + environment: ${{ inputs.environment || 'staging' }} + cluster: ${{ inputs.environment || 'staging' }} + service-name: "boxel-actions-collector-${{ inputs.environment || 'staging' }}" image: ${{ needs.build-actions-collector.outputs.image }} wait-for-service-stability: false From 678be46895e884391b537e8658e9410a6968f534 Mon Sep 17 00:00:00 2001 From: Buck Doyle Date: Wed, 19 Aug 2026 17:43:24 -0400 Subject: [PATCH 05/15] Say which refusal a failed GitHub request was GitHub returns 403 for a rejected credential, for a token whose permissions do not cover the resource, and for an exhausted rate limit alike, distinguishing them only in the response body. The emitted error carried just the status, so a collector sitting at 403 said nothing about which of those it was. Carrying a bounded slice of the body and the remaining request budget into the error puts that in the collector-error line, where the dashboard already shows it. Co-Authored-By: Claude Opus 5 --- .../observability/collectors/actions-queue.ts | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/packages/observability/collectors/actions-queue.ts b/packages/observability/collectors/actions-queue.ts index d290ab3b35a..d23d744bc44 100644 --- a/packages/observability/collectors/actions-queue.ts +++ b/packages/observability/collectors/actions-queue.ts @@ -144,7 +144,23 @@ async function gh(url: string, token: string): Promise { rateLimitRemaining = Number(remaining); } if (!res.ok) { - throw new Error(`GET ${url} → ${res.status} ${res.statusText}`); + // GitHub distinguishes its refusals only in the response body: a rejected + // credential, a token whose repository access omits this one, and an + // exhausted rate limit are all 403. Carrying the body and the remaining + // budget into the error means the emitted collector-error line says which, + // rather than leaving it to be inferred. + let detail = ''; + try { + detail = (await res.text()).slice(0, 300).replace(/\s+/g, ' ').trim(); + } catch { + // A body that cannot be read is not worth failing over; the status still + // carries most of the signal. + } + throw new Error( + `GET ${url} → ${res.status} ${res.statusText}` + + (detail ? ` — ${detail}` : '') + + ` (rate limit remaining: ${res.headers.get('x-ratelimit-remaining') ?? 'unknown'})`, + ); } return (await res.json()) as T; } From ad53cc2d98b975730b3beaa2004eabedddd687ab Mon Sep 17 00:00:00 2001 From: Buck Doyle Date: Wed, 19 Aug 2026 19:26:40 -0400 Subject: [PATCH 06/15] Collapse the dashboard's series and name who owns a stuck job MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every metric panel drew one series per sample rather than one line. `| json` promotes each field of a record to a label, so successive samples never shared a label set and a range aggregation without an explicit `by ()` grouped none of them together. The stat tiles showed a row of numbers instead of a value, and the panels reading per-job fields returned so many series they rendered as no data at all. Every one of these queries has now been run against staging Loki: each returns a single series, and the grouped panels return one per group. The long-running steps table gains the actor, since the first question asked of a job that has sat in one step for hours is whose it is. The temporary push trigger goes with them, having served its purpose — the collector is deployed and reporting, which is what let these queries be checked against real data rather than merged on the assumption they were right. Co-Authored-By: Claude Opus 5 --- .../manual-deploy-actions-collector.yml | 7 ------ .../boxel-status/actions-queue.json | 22 +++++++++---------- 2 files changed, 11 insertions(+), 18 deletions(-) diff --git a/.github/workflows/manual-deploy-actions-collector.yml b/.github/workflows/manual-deploy-actions-collector.yml index d303bcc08ff..22de91e89e2 100644 --- a/.github/workflows/manual-deploy-actions-collector.yml +++ b/.github/workflows/manual-deploy-actions-collector.yml @@ -14,13 +14,6 @@ on: description: Deployment environment required: false default: staging - # TEMPORARY — remove before merge. A workflow_dispatch workflow can only be - # dispatched once its file is on the default branch, so until this merges - # there is no way to build the collector, and without a running collector the - # dashboard on this PR previews with every panel empty. This trigger closes - # that loop so the queries can be checked against real data before merging. - push: - branches: [cs-12571-make-the-github-actions-backlog-visible-in-grafana] permissions: contents: read diff --git a/packages/observability/grafanactl/resources/dashboards/boxel-status/actions-queue.json b/packages/observability/grafanactl/resources/dashboards/boxel-status/actions-queue.json index 298a7eae2fa..f0f2b47813b 100644 --- a/packages/observability/grafanactl/resources/dashboards/boxel-status/actions-queue.json +++ b/packages/observability/grafanactl/resources/dashboards/boxel-status/actions-queue.json @@ -77,7 +77,7 @@ "uid": "loki" }, "editorMode": "code", - "expr": "last_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"snapshot\" | unwrap queued_jobs [$__interval])", + "expr": "last_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"snapshot\" | unwrap queued_jobs [$__interval]) by ()", "refId": "A", "queryType": "range", "legendFormat": "queued" @@ -127,7 +127,7 @@ "uid": "loki" }, "editorMode": "code", - "expr": "last_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"snapshot\" | unwrap running_jobs [$__interval])", + "expr": "last_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"snapshot\" | unwrap running_jobs [$__interval]) by ()", "refId": "A", "queryType": "range", "legendFormat": "running" @@ -177,7 +177,7 @@ "uid": "loki" }, "editorMode": "code", - "expr": "last_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"snapshot\" | unwrap active_runs [$__interval])", + "expr": "last_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"snapshot\" | unwrap active_runs [$__interval]) by ()", "refId": "A", "queryType": "range", "legendFormat": "runs" @@ -227,7 +227,7 @@ "uid": "loki" }, "editorMode": "code", - "expr": "max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"snapshot\" | unwrap queued_jobs [$__interval])", + "expr": "max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"snapshot\" | unwrap queued_jobs [$__interval]) by ()", "refId": "A", "queryType": "range", "legendFormat": "queued" @@ -238,7 +238,7 @@ "uid": "loki" }, "editorMode": "code", - "expr": "max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"snapshot\" | unwrap running_jobs [$__interval])", + "expr": "max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"snapshot\" | unwrap running_jobs [$__interval]) by ()", "refId": "B", "queryType": "range", "legendFormat": "running" @@ -274,7 +274,7 @@ "uid": "loki" }, "editorMode": "code", - "expr": "max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"job\" | status=\"queued\" | unwrap queued_seconds [$__interval])", + "expr": "max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"job\" | status=\"queued\" | unwrap queued_seconds [$__interval]) by ()", "refId": "A", "queryType": "range", "legendFormat": "longest wait" @@ -338,7 +338,7 @@ "uid": "loki" }, "editorMode": "code", - "expr": "quantile_over_time(0.5, {service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"job\" | status=\"queued\" | unwrap queued_seconds [$__interval])", + "expr": "quantile_over_time(0.5, {service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"job\" | status=\"queued\" | unwrap queued_seconds [$__interval]) by ()", "refId": "A", "queryType": "range", "legendFormat": "p50" @@ -349,7 +349,7 @@ "uid": "loki" }, "editorMode": "code", - "expr": "quantile_over_time(0.9, {service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"job\" | status=\"queued\" | unwrap queued_seconds [$__interval])", + "expr": "quantile_over_time(0.9, {service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"job\" | status=\"queued\" | unwrap queued_seconds [$__interval]) by ()", "refId": "B", "queryType": "range", "legendFormat": "p90" @@ -360,7 +360,7 @@ "uid": "loki" }, "editorMode": "code", - "expr": "max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"job\" | status=\"queued\" | unwrap queued_seconds [$__interval])", + "expr": "max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"job\" | status=\"queued\" | unwrap queued_seconds [$__interval]) by ()", "refId": "C", "queryType": "range", "legendFormat": "max" @@ -639,7 +639,7 @@ "uid": "loki" }, "editorMode": "code", - "expr": "topk(25, max by (job, head_branch, workflow, current_step) (last_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"job\" | status=\"in_progress\" | unwrap current_step_seconds [10m])))", + "expr": "topk(25, max by (job, head_branch, workflow, actor, current_step) (last_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"job\" | status=\"in_progress\" | unwrap current_step_seconds [10m])))", "refId": "A", "queryType": "instant" } @@ -688,7 +688,7 @@ "uid": "loki" }, "editorMode": "code", - "expr": "min_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"snapshot\" | unwrap rate_limit_remaining [$__interval])", + "expr": "min_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"snapshot\" | unwrap rate_limit_remaining [$__interval]) by ()", "refId": "A", "queryType": "range", "legendFormat": "remaining" From a9295fed96e7585e336b227f3bd7b1c2959d86e9 Mon Sep 17 00:00:00 2001 From: Buck Doyle Date: Wed, 19 Aug 2026 19:49:08 -0400 Subject: [PATCH 07/15] Rank the dashboard legends by value MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The grouped panels drew a series per branch and listed them alphabetically, so the branch holding most of the queue could sit below the fold while smaller ones were named above it — the panel answered "which branches have queued jobs" when the question being asked of it is "which branch has the most". The legend is now a table sorted by last value, showing last and max, and each grouped query is bounded to the ten largest series at any instant so a busy hour cannot push the ones that matter out of view. Co-Authored-By: Claude Opus 5 --- .../boxel-status/actions-queue.json | 174 ++++++++++++++++-- 1 file changed, 159 insertions(+), 15 deletions(-) diff --git a/packages/observability/grafanactl/resources/dashboards/boxel-status/actions-queue.json b/packages/observability/grafanactl/resources/dashboards/boxel-status/actions-queue.json index f0f2b47813b..f25c45b732d 100644 --- a/packages/observability/grafanactl/resources/dashboards/boxel-status/actions-queue.json +++ b/packages/observability/grafanactl/resources/dashboards/boxel-status/actions-queue.json @@ -250,7 +250,23 @@ }, "overrides": [] }, - "options": {}, + "options": { + "legend": { + "displayMode": "table", + "placement": "bottom", + "showLegend": true, + "calcs": [ + "lastNotNull", + "max" + ], + "sortBy": "Last *", + "sortDesc": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, "description": "Queued depth rising while running stays flat means the pool is saturated, not slow." }, { @@ -373,7 +389,23 @@ }, "overrides": [] }, - "options": {}, + "options": { + "legend": { + "displayMode": "table", + "placement": "bottom", + "showLegend": true, + "calcs": [ + "lastNotNull", + "max" + ], + "sortBy": "Last *", + "sortDesc": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, "description": "How long jobs have been waiting, measured from job creation." }, { @@ -410,7 +442,7 @@ "uid": "loki" }, "editorMode": "code", - "expr": "sum by (key) (max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"group\" | dimension=\"branch\" | unwrap queued [$__interval]))", + "expr": "topk(10, sum by (key) (max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"group\" | dimension=\"branch\" | unwrap queued [$__interval])))", "refId": "A", "queryType": "range", "legendFormat": "{{key}}" @@ -422,7 +454,23 @@ }, "overrides": [] }, - "options": {}, + "options": { + "legend": { + "displayMode": "table", + "placement": "bottom", + "showLegend": true, + "calcs": [ + "lastNotNull", + "max" + ], + "sortBy": "Last *", + "sortDesc": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, "description": "Queued jobs grouped by branch, from the collector's per-group snapshot lines." }, { @@ -446,7 +494,7 @@ "uid": "loki" }, "editorMode": "code", - "expr": "sum by (key) (max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"group\" | dimension=\"actor\" | unwrap queued [$__interval]))", + "expr": "topk(10, sum by (key) (max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"group\" | dimension=\"actor\" | unwrap queued [$__interval])))", "refId": "A", "queryType": "range", "legendFormat": "{{key}}" @@ -458,7 +506,23 @@ }, "overrides": [] }, - "options": {}, + "options": { + "legend": { + "displayMode": "table", + "placement": "bottom", + "showLegend": true, + "calcs": [ + "lastNotNull", + "max" + ], + "sortBy": "Last *", + "sortDesc": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, "description": "Queued jobs grouped by actor, from the collector's per-group snapshot lines." }, { @@ -482,7 +546,7 @@ "uid": "loki" }, "editorMode": "code", - "expr": "sum by (key) (max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"group\" | dimension=\"workflow\" | unwrap queued [$__interval]))", + "expr": "topk(10, sum by (key) (max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"group\" | dimension=\"workflow\" | unwrap queued [$__interval])))", "refId": "A", "queryType": "range", "legendFormat": "{{key}}" @@ -494,7 +558,23 @@ }, "overrides": [] }, - "options": {}, + "options": { + "legend": { + "displayMode": "table", + "placement": "bottom", + "showLegend": true, + "calcs": [ + "lastNotNull", + "max" + ], + "sortBy": "Last *", + "sortDesc": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, "description": "Queued jobs grouped by workflow, from the collector's per-group snapshot lines." }, { @@ -518,7 +598,7 @@ "uid": "loki" }, "editorMode": "code", - "expr": "sum by (key) (max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"group\" | dimension=\"branch\" | unwrap running [$__interval]))", + "expr": "topk(10, sum by (key) (max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"group\" | dimension=\"branch\" | unwrap running [$__interval])))", "refId": "A", "queryType": "range", "legendFormat": "{{key}}" @@ -530,7 +610,23 @@ }, "overrides": [] }, - "options": {}, + "options": { + "legend": { + "displayMode": "table", + "placement": "bottom", + "showLegend": true, + "calcs": [ + "lastNotNull", + "max" + ], + "sortBy": "Last *", + "sortDesc": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, "description": "Runners currently held, grouped by branch." }, { @@ -554,7 +650,7 @@ "uid": "loki" }, "editorMode": "code", - "expr": "sum by (key) (max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"group\" | dimension=\"actor\" | unwrap running [$__interval]))", + "expr": "topk(10, sum by (key) (max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"group\" | dimension=\"actor\" | unwrap running [$__interval])))", "refId": "A", "queryType": "range", "legendFormat": "{{key}}" @@ -566,7 +662,23 @@ }, "overrides": [] }, - "options": {}, + "options": { + "legend": { + "displayMode": "table", + "placement": "bottom", + "showLegend": true, + "calcs": [ + "lastNotNull", + "max" + ], + "sortBy": "Last *", + "sortDesc": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, "description": "Runners currently held, grouped by actor." }, { @@ -590,7 +702,7 @@ "uid": "loki" }, "editorMode": "code", - "expr": "sum by (key) (max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"group\" | dimension=\"workflow\" | unwrap running [$__interval]))", + "expr": "topk(10, sum by (key) (max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"group\" | dimension=\"workflow\" | unwrap running [$__interval])))", "refId": "A", "queryType": "range", "legendFormat": "{{key}}" @@ -602,7 +714,23 @@ }, "overrides": [] }, - "options": {}, + "options": { + "legend": { + "displayMode": "table", + "placement": "bottom", + "showLegend": true, + "calcs": [ + "lastNotNull", + "max" + ], + "sortBy": "Last *", + "sortDesc": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, "description": "Runners currently held, grouped by workflow." }, { @@ -700,7 +828,23 @@ }, "overrides": [] }, - "options": {}, + "options": { + "legend": { + "displayMode": "table", + "placement": "bottom", + "showLegend": true, + "calcs": [ + "lastNotNull", + "max" + ], + "sortBy": "Last *", + "sortDesc": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, "description": "Sampling stops rather than reporting partial depth when this nears the reserve." }, { From f0c2c33d64816f00d5c974b6f94f1da5dc14ed5b Mon Sep 17 00:00:00 2001 From: Buck Doyle Date: Wed, 19 Aug 2026 20:12:19 -0400 Subject: [PATCH 08/15] Stop the dashboard drawing spikes where it should draw lines MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The collector samples every two minutes, but Grafana derives $__interval from panel width and on a multi-hour window that lands well under the sample period. Most range windows therefore covered no sample at all — at thirty seconds the queries return nothing — so the panels rendered as isolated spikes rather than a series, and the wait panels in particular looked like noise. Flooring the panel interval at five minutes keeps every window covering at least one sample, and spanning nulls bridges a gap where a sample is missed rather than breaking the line. Every query was re-run against staging Loki at the floored interval: each returns a continuous series where it previously returned a handful of scattered points. Co-Authored-By: Claude Opus 5 --- .../boxel-status/actions-queue.json | 75 +++++++++++++------ 1 file changed, 53 insertions(+), 22 deletions(-) diff --git a/packages/observability/grafanactl/resources/dashboards/boxel-status/actions-queue.json b/packages/observability/grafanactl/resources/dashboards/boxel-status/actions-queue.json index f25c45b732d..b89688e9b79 100644 --- a/packages/observability/grafanactl/resources/dashboards/boxel-status/actions-queue.json +++ b/packages/observability/grafanactl/resources/dashboards/boxel-status/actions-queue.json @@ -104,7 +104,8 @@ "textMode": "auto", "wideLayout": true }, - "description": "Jobs waiting for a runner at the most recent sample." + "description": "Jobs waiting for a runner at the most recent sample.", + "interval": "5m" }, { "id": 4, @@ -154,7 +155,8 @@ "textMode": "auto", "wideLayout": true }, - "description": "Jobs currently holding a runner." + "description": "Jobs currently holding a runner.", + "interval": "5m" }, { "id": 5, @@ -204,7 +206,8 @@ "textMode": "auto", "wideLayout": true }, - "description": "Workflow runs in queued or in_progress state." + "description": "Workflow runs in queued or in_progress state.", + "interval": "5m" }, { "id": 6, @@ -246,7 +249,9 @@ ], "fieldConfig": { "defaults": { - "custom": {} + "custom": { + "spanNulls": true + } }, "overrides": [] }, @@ -267,7 +272,8 @@ "sort": "desc" } }, - "description": "Queued depth rising while running stays flat means the pool is saturated, not slow." + "description": "Queued depth rising while running stays flat means the pool is saturated, not slow.", + "interval": "5m" }, { "id": 7, @@ -318,7 +324,8 @@ "textMode": "auto", "wideLayout": true }, - "description": "Age of the oldest job still waiting for a runner." + "description": "Age of the oldest job still waiting for a runner.", + "interval": "5m" }, { "id": 8, @@ -384,7 +391,9 @@ ], "fieldConfig": { "defaults": { - "custom": {}, + "custom": { + "spanNulls": true + }, "unit": "s" }, "overrides": [] @@ -406,7 +415,8 @@ "sort": "desc" } }, - "description": "How long jobs have been waiting, measured from job creation." + "description": "How long jobs have been waiting, measured from job creation.", + "interval": "5m" }, { "id": 10, @@ -450,7 +460,9 @@ ], "fieldConfig": { "defaults": { - "custom": {} + "custom": { + "spanNulls": true + } }, "overrides": [] }, @@ -471,7 +483,8 @@ "sort": "desc" } }, - "description": "Queued jobs grouped by branch, from the collector's per-group snapshot lines." + "description": "Queued jobs grouped by branch, from the collector's per-group snapshot lines.", + "interval": "5m" }, { "id": 12, @@ -502,7 +515,9 @@ ], "fieldConfig": { "defaults": { - "custom": {} + "custom": { + "spanNulls": true + } }, "overrides": [] }, @@ -523,7 +538,8 @@ "sort": "desc" } }, - "description": "Queued jobs grouped by actor, from the collector's per-group snapshot lines." + "description": "Queued jobs grouped by actor, from the collector's per-group snapshot lines.", + "interval": "5m" }, { "id": 13, @@ -554,7 +570,9 @@ ], "fieldConfig": { "defaults": { - "custom": {} + "custom": { + "spanNulls": true + } }, "overrides": [] }, @@ -575,7 +593,8 @@ "sort": "desc" } }, - "description": "Queued jobs grouped by workflow, from the collector's per-group snapshot lines." + "description": "Queued jobs grouped by workflow, from the collector's per-group snapshot lines.", + "interval": "5m" }, { "id": 14, @@ -606,7 +625,9 @@ ], "fieldConfig": { "defaults": { - "custom": {} + "custom": { + "spanNulls": true + } }, "overrides": [] }, @@ -627,7 +648,8 @@ "sort": "desc" } }, - "description": "Runners currently held, grouped by branch." + "description": "Runners currently held, grouped by branch.", + "interval": "5m" }, { "id": 15, @@ -658,7 +680,9 @@ ], "fieldConfig": { "defaults": { - "custom": {} + "custom": { + "spanNulls": true + } }, "overrides": [] }, @@ -679,7 +703,8 @@ "sort": "desc" } }, - "description": "Runners currently held, grouped by actor." + "description": "Runners currently held, grouped by actor.", + "interval": "5m" }, { "id": 16, @@ -710,7 +735,9 @@ ], "fieldConfig": { "defaults": { - "custom": {} + "custom": { + "spanNulls": true + } }, "overrides": [] }, @@ -731,7 +758,8 @@ "sort": "desc" } }, - "description": "Runners currently held, grouped by workflow." + "description": "Runners currently held, grouped by workflow.", + "interval": "5m" }, { "id": 17, @@ -824,7 +852,9 @@ ], "fieldConfig": { "defaults": { - "custom": {} + "custom": { + "spanNulls": true + } }, "overrides": [] }, @@ -845,7 +875,8 @@ "sort": "desc" } }, - "description": "Sampling stops rather than reporting partial depth when this nears the reserve." + "description": "Sampling stops rather than reporting partial depth when this nears the reserve.", + "interval": "5m" }, { "id": 21, From b2a295d15fcb15900223f03088f75a9c6864da4a Mon Sep 17 00:00:00 2001 From: Buck Doyle Date: Wed, 19 Aug 2026 20:35:56 -0400 Subject: [PATCH 09/15] Put the dashboard's explanation after what it explains The description occupied the top of the page, so opening the dashboard during an incident meant scrolling past prose to reach the queue depth. It reads better as an appendix: useful once, when someone wants to know where the numbers come from, and in the way every other time. Co-Authored-By: Claude Opus 5 --- .../boxel-status/actions-queue.json | 92 +++++++++---------- 1 file changed, 46 insertions(+), 46 deletions(-) diff --git a/packages/observability/grafanactl/resources/dashboards/boxel-status/actions-queue.json b/packages/observability/grafanactl/resources/dashboards/boxel-status/actions-queue.json index b89688e9b79..30857d10dbc 100644 --- a/packages/observability/grafanactl/resources/dashboards/boxel-status/actions-queue.json +++ b/packages/observability/grafanactl/resources/dashboards/boxel-status/actions-queue.json @@ -17,32 +17,6 @@ "graphTooltip": 0, "links": [], "panels": [ - { - "id": 1, - "type": "text", - "title": "About this dashboard", - "datasource": { - "type": "loki", - "uid": "loki" - }, - "gridPos": { - "x": 0, - "y": 0, - "w": 24, - "h": 4 - }, - "targets": [], - "fieldConfig": { - "defaults": { - "custom": {} - }, - "overrides": [] - }, - "options": { - "mode": "markdown", - "content": "Sampled from the GitHub Actions REST API by `packages/observability/collectors/actions-queue.ts`, which emits one JSON line per observation on channel `boxel:actions-queue`.\n\nThe collector runs **outside** GitHub Actions on purpose — a scheduled workflow would queue behind the backlog it measures and go blind during the incident it exists for.\n\nDepth and grouped depth come from periodic snapshots, so each point is a sample rather than a continuous measure; gaps mean the collector was down or throttled (see *Collector health*)." - } - }, { "id": 2, "type": "row", @@ -50,7 +24,7 @@ "collapsed": false, "gridPos": { "x": 0, - "y": 4, + "y": 0, "w": 24, "h": 1 }, @@ -66,7 +40,7 @@ }, "gridPos": { "x": 0, - "y": 5, + "y": 1, "w": 4, "h": 4 }, @@ -117,7 +91,7 @@ }, "gridPos": { "x": 4, - "y": 5, + "y": 1, "w": 4, "h": 4 }, @@ -168,7 +142,7 @@ }, "gridPos": { "x": 8, - "y": 5, + "y": 1, "w": 4, "h": 4 }, @@ -219,7 +193,7 @@ }, "gridPos": { "x": 12, - "y": 5, + "y": 1, "w": 12, "h": 8 }, @@ -285,7 +259,7 @@ }, "gridPos": { "x": 0, - "y": 9, + "y": 5, "w": 12, "h": 4 }, @@ -334,7 +308,7 @@ "collapsed": false, "gridPos": { "x": 0, - "y": 13, + "y": 9, "w": 24, "h": 1 }, @@ -350,7 +324,7 @@ }, "gridPos": { "x": 0, - "y": 14, + "y": 10, "w": 24, "h": 8 }, @@ -425,7 +399,7 @@ "collapsed": false, "gridPos": { "x": 0, - "y": 22, + "y": 18, "w": 24, "h": 1 }, @@ -441,7 +415,7 @@ }, "gridPos": { "x": 0, - "y": 23, + "y": 19, "w": 8, "h": 8 }, @@ -496,7 +470,7 @@ }, "gridPos": { "x": 8, - "y": 23, + "y": 19, "w": 8, "h": 8 }, @@ -551,7 +525,7 @@ }, "gridPos": { "x": 16, - "y": 23, + "y": 19, "w": 8, "h": 8 }, @@ -606,7 +580,7 @@ }, "gridPos": { "x": 0, - "y": 31, + "y": 27, "w": 8, "h": 8 }, @@ -661,7 +635,7 @@ }, "gridPos": { "x": 8, - "y": 31, + "y": 27, "w": 8, "h": 8 }, @@ -716,7 +690,7 @@ }, "gridPos": { "x": 16, - "y": 31, + "y": 27, "w": 8, "h": 8 }, @@ -768,7 +742,7 @@ "collapsed": false, "gridPos": { "x": 0, - "y": 39, + "y": 35, "w": 24, "h": 1 }, @@ -784,7 +758,7 @@ }, "gridPos": { "x": 0, - "y": 40, + "y": 36, "w": 24, "h": 10 }, @@ -817,7 +791,7 @@ "collapsed": false, "gridPos": { "x": 0, - "y": 50, + "y": 46, "w": 24, "h": 1 }, @@ -833,7 +807,7 @@ }, "gridPos": { "x": 0, - "y": 51, + "y": 47, "w": 12, "h": 6 }, @@ -888,7 +862,7 @@ }, "gridPos": { "x": 12, - "y": 51, + "y": 47, "w": 12, "h": 6 }, @@ -912,6 +886,32 @@ }, "options": {}, "description": "Gaps in the series above should have a matching line here." + }, + { + "id": 1, + "type": "text", + "title": "About this dashboard", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "gridPos": { + "x": 0, + "y": 53, + "w": 24, + "h": 4 + }, + "targets": [], + "fieldConfig": { + "defaults": { + "custom": {} + }, + "overrides": [] + }, + "options": { + "mode": "markdown", + "content": "Sampled from the GitHub Actions REST API by `packages/observability/collectors/actions-queue.ts`, which emits one JSON line per observation on channel `boxel:actions-queue`.\n\nThe collector runs **outside** GitHub Actions on purpose — a scheduled workflow would queue behind the backlog it measures and go blind during the incident it exists for.\n\nDepth and grouped depth come from periodic snapshots, so each point is a sample rather than a continuous measure; gaps mean the collector was down or throttled (see *Collector health*)." + } } ], "refresh": "1m", From b4964e0a0d7cdc39ee7bf282e5983d5e8552d90f Mon Sep 17 00:00:00 2001 From: Buck Doyle Date: Wed, 19 Aug 2026 20:39:07 -0400 Subject: [PATCH 10/15] Add point-in-time tables beside the queue-depth history MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The grouped panels are timeseries over the dashboard's window, so a branch that was queue-heavy an hour ago keeps a line long after its jobs have gone. Read at a glance during an incident that says "this branch has the most queued", when what it means is "this branch had the most queued at some point in the last six hours" — twice now that has sent someone looking for jobs that had already finished. A "Right now" row of instant queries answers the other question directly: what is queued, and who is holding runners, at this moment. Rows with nothing queued are filtered out, so a branch drops off the table when its work is done rather than lingering at zero. The timeseries stay. "How did we get here" and "what is true now" are different questions and the history is the right shape for the first. Co-Authored-By: Claude Opus 5 --- .../boxel-status/actions-queue.json | 232 ++++++++++++++++-- 1 file changed, 217 insertions(+), 15 deletions(-) diff --git a/packages/observability/grafanactl/resources/dashboards/boxel-status/actions-queue.json b/packages/observability/grafanactl/resources/dashboards/boxel-status/actions-queue.json index 30857d10dbc..4f674e04b32 100644 --- a/packages/observability/grafanactl/resources/dashboards/boxel-status/actions-queue.json +++ b/packages/observability/grafanactl/resources/dashboards/boxel-status/actions-queue.json @@ -308,7 +308,7 @@ "collapsed": false, "gridPos": { "x": 0, - "y": 9, + "y": 18, "w": 24, "h": 1 }, @@ -324,7 +324,7 @@ }, "gridPos": { "x": 0, - "y": 10, + "y": 19, "w": 24, "h": 8 }, @@ -399,7 +399,7 @@ "collapsed": false, "gridPos": { "x": 0, - "y": 18, + "y": 27, "w": 24, "h": 1 }, @@ -415,7 +415,7 @@ }, "gridPos": { "x": 0, - "y": 19, + "y": 28, "w": 8, "h": 8 }, @@ -470,7 +470,7 @@ }, "gridPos": { "x": 8, - "y": 19, + "y": 28, "w": 8, "h": 8 }, @@ -525,7 +525,7 @@ }, "gridPos": { "x": 16, - "y": 19, + "y": 28, "w": 8, "h": 8 }, @@ -580,7 +580,7 @@ }, "gridPos": { "x": 0, - "y": 27, + "y": 36, "w": 8, "h": 8 }, @@ -635,7 +635,7 @@ }, "gridPos": { "x": 8, - "y": 27, + "y": 36, "w": 8, "h": 8 }, @@ -690,7 +690,7 @@ }, "gridPos": { "x": 16, - "y": 27, + "y": 36, "w": 8, "h": 8 }, @@ -742,7 +742,7 @@ "collapsed": false, "gridPos": { "x": 0, - "y": 35, + "y": 44, "w": 24, "h": 1 }, @@ -758,7 +758,7 @@ }, "gridPos": { "x": 0, - "y": 36, + "y": 45, "w": 24, "h": 10 }, @@ -791,7 +791,7 @@ "collapsed": false, "gridPos": { "x": 0, - "y": 46, + "y": 55, "w": 24, "h": 1 }, @@ -807,7 +807,7 @@ }, "gridPos": { "x": 0, - "y": 47, + "y": 56, "w": 12, "h": 6 }, @@ -862,7 +862,7 @@ }, "gridPos": { "x": 12, - "y": 47, + "y": 56, "w": 12, "h": 6 }, @@ -897,7 +897,7 @@ }, "gridPos": { "x": 0, - "y": 53, + "y": 62, "w": 24, "h": 4 }, @@ -912,6 +912,208 @@ "mode": "markdown", "content": "Sampled from the GitHub Actions REST API by `packages/observability/collectors/actions-queue.ts`, which emits one JSON line per observation on channel `boxel:actions-queue`.\n\nThe collector runs **outside** GitHub Actions on purpose — a scheduled workflow would queue behind the backlog it measures and go blind during the incident it exists for.\n\nDepth and grouped depth come from periodic snapshots, so each point is a sample rather than a continuous measure; gaps mean the collector was down or throttled (see *Collector health*)." } + }, + { + "id": 40, + "type": "row", + "title": "Right now", + "collapsed": false, + "gridPos": { + "x": 0, + "y": 9, + "w": 24, + "h": 1 + }, + "panels": [] + }, + { + "id": 41, + "type": "table", + "title": "Queued now, by branch", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "description": "A point-in-time count, unlike the timeseries above — a branch whose jobs have finished disappears from this table instead of leaving a line behind.", + "gridPos": { + "x": 0, + "y": 10, + "w": 8, + "h": 8 + }, + "interval": "5m", + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "editorMode": "code", + "refId": "A", + "queryType": "instant", + "expr": "sum by (key) (last_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"group\" | dimension=\"branch\" | unwrap queued [5m])) > 0" + } + ], + "transformations": [ + { + "id": "organize", + "options": { + "excludeByName": { + "Time": true + }, + "renameByName": { + "key": "branch", + "Value": "jobs", + "Value #A": "jobs" + } + } + }, + { + "id": "sortBy", + "options": { + "fields": {}, + "sort": [ + { + "field": "jobs", + "desc": true + } + ] + } + } + ], + "fieldConfig": { + "defaults": { + "custom": {} + }, + "overrides": [] + }, + "options": {} + }, + { + "id": 42, + "type": "table", + "title": "Runners held now, by branch", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "description": "Which branches are occupying the pool at this moment.", + "gridPos": { + "x": 8, + "y": 10, + "w": 8, + "h": 8 + }, + "interval": "5m", + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "editorMode": "code", + "refId": "A", + "queryType": "instant", + "expr": "sum by (key) (last_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"group\" | dimension=\"branch\" | unwrap running [5m])) > 0" + } + ], + "transformations": [ + { + "id": "organize", + "options": { + "excludeByName": { + "Time": true + }, + "renameByName": { + "key": "branch", + "Value": "jobs", + "Value #A": "jobs" + } + } + }, + { + "id": "sortBy", + "options": { + "fields": {}, + "sort": [ + { + "field": "jobs", + "desc": true + } + ] + } + } + ], + "fieldConfig": { + "defaults": { + "custom": {} + }, + "overrides": [] + }, + "options": {} + }, + { + "id": 43, + "type": "table", + "title": "Queued now, by author", + "datasource": { + "type": "loki", + "uid": "loki" + }, + "description": "The same instant count grouped by whoever triggered the run.", + "gridPos": { + "x": 16, + "y": 10, + "w": 8, + "h": 8 + }, + "interval": "5m", + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "editorMode": "code", + "refId": "A", + "queryType": "instant", + "expr": "sum by (key) (last_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"group\" | dimension=\"actor\" | unwrap queued [5m])) > 0" + } + ], + "transformations": [ + { + "id": "organize", + "options": { + "excludeByName": { + "Time": true + }, + "renameByName": { + "key": "author", + "Value": "jobs", + "Value #A": "jobs" + } + } + }, + { + "id": "sortBy", + "options": { + "fields": {}, + "sort": [ + { + "field": "jobs", + "desc": true + } + ] + } + } + ], + "fieldConfig": { + "defaults": { + "custom": {} + }, + "overrides": [] + }, + "options": {} } ], "refresh": "1m", From e041fbf6a29f2f9f7600d7d846154850eb2eef2e Mon Sep 17 00:00:00 2001 From: Buck Doyle Date: Wed, 19 Aug 2026 21:01:56 -0400 Subject: [PATCH 11/15] Make the dashboard's table rows link into GitHub MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Seeing that a job has sat in one step for four hours raises exactly one question, and answering it meant copying a job name into the Actions search by hand. The rows now carry links on the text already in them, so no column shows a URL. A job opens its own page on GitHub, which needs the run and job identifiers, so the query carries them and the transformation hides the columns — they are identifiers rather than information. A branch opens the Actions runs filtered to it, and an author their profile. Every link was resolved against a live row and fetched; all six return 200. Co-Authored-By: Claude Opus 5 --- .../boxel-status/actions-queue.json | 140 +++++++++++++++++- 1 file changed, 134 insertions(+), 6 deletions(-) diff --git a/packages/observability/grafanactl/resources/dashboards/boxel-status/actions-queue.json b/packages/observability/grafanactl/resources/dashboards/boxel-status/actions-queue.json index 4f674e04b32..6c52406f00b 100644 --- a/packages/observability/grafanactl/resources/dashboards/boxel-status/actions-queue.json +++ b/packages/observability/grafanactl/resources/dashboards/boxel-status/actions-queue.json @@ -769,7 +769,7 @@ "uid": "loki" }, "editorMode": "code", - "expr": "topk(25, max by (job, head_branch, workflow, actor, current_step) (last_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"job\" | status=\"in_progress\" | unwrap current_step_seconds [10m])))", + "expr": "topk(25, max by (job, head_branch, workflow, actor, current_step, run_id, job_id) (last_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"job\" | status=\"in_progress\" | unwrap current_step_seconds [10m])))", "refId": "A", "queryType": "instant" } @@ -779,10 +779,81 @@ "custom": {}, "unit": "s" }, - "overrides": [] + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "job" + }, + "properties": [ + { + "id": "links", + "value": [ + { + "title": "Open this job on GitHub", + "url": "https://github.com/cardstack/boxel/actions/runs/${__data.fields.run_id}/job/${__data.fields.job_id}", + "targetBlank": true + } + ] + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "head_branch" + }, + "properties": [ + { + "id": "links", + "value": [ + { + "title": "Actions runs for this branch", + "url": "https://github.com/cardstack/boxel/actions?query=branch%3A${__value.raw}", + "targetBlank": true + } + ] + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "actor" + }, + "properties": [ + { + "id": "links", + "value": [ + { + "title": "GitHub profile", + "url": "https://github.com/${__value.raw}", + "targetBlank": true + } + ] + } + ] + } + ] }, "options": {}, - "description": "A job whose current step has run far longer than its peers is the wedge signal — compare against the same step on other shards before concluding it is stuck." + "description": "A job whose current step has run far longer than its peers is the wedge signal — compare against the same step on other shards before concluding it is stuck.", + "transformations": [ + { + "id": "organize", + "options": { + "excludeByName": { + "Time": true, + "run_id": true, + "job_id": true + }, + "renameByName": { + "Value": "in step", + "Value #A": "in step" + } + } + } + ] }, { "id": 19, @@ -985,7 +1056,26 @@ "defaults": { "custom": {} }, - "overrides": [] + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "branch" + }, + "properties": [ + { + "id": "links", + "value": [ + { + "title": "Actions runs for this branch", + "url": "https://github.com/cardstack/boxel/actions?query=branch%3A${__value.raw}", + "targetBlank": true + } + ] + } + ] + } + ] }, "options": {} }, @@ -1048,7 +1138,26 @@ "defaults": { "custom": {} }, - "overrides": [] + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "branch" + }, + "properties": [ + { + "id": "links", + "value": [ + { + "title": "Actions runs for this branch", + "url": "https://github.com/cardstack/boxel/actions?query=branch%3A${__value.raw}", + "targetBlank": true + } + ] + } + ] + } + ] }, "options": {} }, @@ -1111,7 +1220,26 @@ "defaults": { "custom": {} }, - "overrides": [] + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "author" + }, + "properties": [ + { + "id": "links", + "value": [ + { + "title": "GitHub profile", + "url": "https://github.com/${__value.raw}", + "targetBlank": true + } + ] + } + ] + } + ] }, "options": {} } From e0e92f9c5db723f397dcabdab3dd21956266e13d Mon Sep 17 00:00:00 2001 From: Buck Doyle Date: Thu, 20 Aug 2026 08:28:13 -0400 Subject: [PATCH 12/15] Abandon a sample rather than publish an incomplete one MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A failed job listing for one active run was logged and skipped, and the grouped totals and snapshot were then emitted regardless. That put a confidently low queue depth on the dashboard at exactly the moment the API was failing — contradicting the rate-limit reserve a few lines above, which abandons a sample for precisely that reason. Now only a 404 continues: a run reaped between the listing and the fetch has no jobs left to count, so omitting it understates nothing. Any other status means this run's jobs are unknown rather than absent, so the sample is abandoned with a collector-aborted line, leaving a visible gap instead of a wrong number. The long-running-steps table narrows its window from ten minutes to four, which bounds how long a finished job's last observation can linger in a table about what is running now. Its description states the residual, because a job that changes step mid-window still contributes a row per step and shortening the window measurably does not change that. Queue depth over time also carries active runs, so each stat tile beside it has a hoverable counterpart — a stat panel has no tooltip of its own. Co-Authored-By: Claude Opus 5 --- .../observability/collectors/actions-queue.ts | 28 +++++++++++++++---- .../boxel-status/actions-queue.json | 17 +++++++++-- 2 files changed, 37 insertions(+), 8 deletions(-) diff --git a/packages/observability/collectors/actions-queue.ts b/packages/observability/collectors/actions-queue.ts index d23d744bc44..2a39c91049b 100644 --- a/packages/observability/collectors/actions-queue.ts +++ b/packages/observability/collectors/actions-queue.ts @@ -156,11 +156,13 @@ async function gh(url: string, token: string): Promise { // A body that cannot be read is not worth failing over; the status still // carries most of the signal. } - throw new Error( + const err = new Error( `GET ${url} → ${res.status} ${res.statusText}` + (detail ? ` — ${detail}` : '') + ` (rate limit remaining: ${res.headers.get('x-ratelimit-remaining') ?? 'unknown'})`, - ); + ) as Error & { status?: number }; + err.status = res.status; + throw err; } return (await res.json()) as T; } @@ -255,8 +257,7 @@ async function sample(opts: Options): Promise { (b) => b.jobs ?? [], ); } catch (e) { - // A run can complete and be reaped between listing and this fetch. That is - // ordinary, so it must not abort the whole sample. + const status = (e as { status?: number }).status; emit({ event_type: 'collector-error', observed_at, @@ -264,7 +265,24 @@ async function sample(opts: Options): Promise { run_id: run.id, message: e instanceof Error ? e.message : String(e), }); - continue; + // A run that completed and was reaped between the listing and this fetch + // answers 404, and has no jobs left to count — omitting it understates + // nothing, so the sample continues. + if (status === 404) continue; + // Anything else means this run's jobs are unknown rather than absent. + // Publishing the totals anyway would put a confidently low queue depth + // on the dashboard at precisely the moment the API is failing, which is + // worse than the visible gap a skipped sample leaves — the same reason + // the rate-limit reserve above abandons the sample rather than trimming + // it. + emit({ + event_type: 'collector-aborted', + observed_at, + repo: opts.repo, + active_runs: runs.length, + reason: `job listing failed for run ${run.id} with status ${status ?? 'unknown'}`, + }); + return; } for (const job of jobs) { diff --git a/packages/observability/grafanactl/resources/dashboards/boxel-status/actions-queue.json b/packages/observability/grafanactl/resources/dashboards/boxel-status/actions-queue.json index 6c52406f00b..8cfee2af7e7 100644 --- a/packages/observability/grafanactl/resources/dashboards/boxel-status/actions-queue.json +++ b/packages/observability/grafanactl/resources/dashboards/boxel-status/actions-queue.json @@ -219,6 +219,17 @@ "refId": "B", "queryType": "range", "legendFormat": "running" + }, + { + "datasource": { + "type": "loki", + "uid": "loki" + }, + "editorMode": "code", + "expr": "max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"snapshot\" | unwrap active_runs [$__interval]) by ()", + "refId": "C", + "queryType": "range", + "legendFormat": "active runs" } ], "fieldConfig": { @@ -246,7 +257,7 @@ "sort": "desc" } }, - "description": "Queued depth rising while running stays flat means the pool is saturated, not slow.", + "description": "Queued depth rising while running stays flat means the pool is saturated rather than slow.\n\nCarries the same three values as the stat tiles beside it, because a stat panel has no hover tooltip — read the headline there, hover here.", "interval": "5m" }, { @@ -769,7 +780,7 @@ "uid": "loki" }, "editorMode": "code", - "expr": "topk(25, max by (job, head_branch, workflow, actor, current_step, run_id, job_id) (last_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"job\" | status=\"in_progress\" | unwrap current_step_seconds [10m])))", + "expr": "topk(25, max by (job, head_branch, workflow, actor, current_step, run_id, job_id) (last_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"job\" | status=\"in_progress\" | unwrap current_step_seconds [4m])))", "refId": "A", "queryType": "instant" } @@ -837,7 +848,7 @@ ] }, "options": {}, - "description": "A job whose current step has run far longer than its peers is the wedge signal — compare against the same step on other shards before concluding it is stuck.", + "description": "Ranked by how long a job's current step has been running. A job whose step has run far longer than the same step on its siblings is the wedge signal — compare before concluding it is stuck.\n\nThe window is four minutes against a two-minute sample interval, so a row can be up to that stale: a job that finished moments ago may still appear, and a job that just changed step may briefly show both. Neither distorts what this panel is for, since a stall worth acting on is measured in hours.", "transformations": [ { "id": "organize", From ada094bbb7982f91fac995d210a132b33bca5480 Mon Sep 17 00:00:00 2001 From: Buck Doyle Date: Thu, 20 Aug 2026 08:58:02 -0400 Subject: [PATCH 13/15] Sample the Actions queue every minute MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Measured over the collector's own history the budget is nowhere near spent: 4 active runs at the median, 43 at p95, 51 at peak, and the remaining request count never fell below 3,792 of 5,000. A minute doubles the resolution and spends about 3,200 an hour at that peak. Thirty seconds does not fit. The cost is one request per active run, so it crosses the hourly limit somewhere around forty concurrent runs — which is precisely when the queue is worth watching closely. The reserve would keep the token alive by skipping samples, meaning the collector would go sparse exactly during an incident. Going finer than a minute needs the per-sample cost reduced, not the interval shortened. Co-Authored-By: Claude Opus 5 --- packages/observability/README.md | 14 ++++++++----- .../observability/collectors/actions-queue.ts | 21 +++++++++++++------ 2 files changed, 24 insertions(+), 11 deletions(-) diff --git a/packages/observability/README.md b/packages/observability/README.md index e8b241e170f..fffc7d01aa0 100644 --- a/packages/observability/README.md +++ b/packages/observability/README.md @@ -86,7 +86,7 @@ Samples GitHub Actions queue depth, per-job wait times and runner consumption. ```bash GITHUB_TOKEN=$(gh auth token) node collectors/actions-queue.ts --once -GITHUB_TOKEN=$(gh auth token) node collectors/actions-queue.ts # loops, default 120s +GITHUB_TOKEN=$(gh auth token) node collectors/actions-queue.ts # loops, default 60s ``` It emits four event types, all on channel `boxel:actions-queue`: @@ -108,10 +108,14 @@ Two constraints are load-bearing: - **It must not run as a scheduled GitHub Actions workflow.** It would queue behind the backlog it measures and go blind during the incident it exists for. Hosted, it belongs on a schedule outside Actions. -- **A sample costs one request per active run, plus two.** At ~60 runs in - flight a 60-second interval would spend most of a token's 5,000 hourly REST - requests, so the default interval is 120s and sampling stops rather than - reporting partial depth when the remaining budget nears its reserve. +- **A sample costs one request per active run, plus two.** The hourly spend + therefore scales with how busy the repository is, against a token's 5,000 + requests an hour. At the observed peak of ~50 concurrent runs a 60-second + interval spends around 3,200 an hour, which is comfortable; 30 seconds + exceeds the budget above roughly forty concurrent runs, so higher resolution + needs the per-sample cost reduced rather than the interval shortened. + Sampling stops rather than reporting partial depth when the remaining budget + nears its reserve. The dashboard reads `{service="actions-collector", env="$env"}`, so a hosted deployment needs to log under that service name. diff --git a/packages/observability/collectors/actions-queue.ts b/packages/observability/collectors/actions-queue.ts index 2a39c91049b..72f4a9670f6 100644 --- a/packages/observability/collectors/actions-queue.ts +++ b/packages/observability/collectors/actions-queue.ts @@ -14,12 +14,21 @@ const CHANNEL = 'boxel:actions-queue'; -// One sample costs two run-list requests plus one per active run. A busy hour -// on this repository has ~60 runs in flight, so a 60-second interval would spend -// ~3,800 of the 5,000 hourly REST requests a token is allowed — enough to -// starve anything else using the same token. Two minutes halves that, and the -// reserve below stops a spike from consuming the rest. -const DEFAULT_INTERVAL_SECONDS = 120; +// One sample costs two run-list requests plus one per active run, so the hourly +// spend scales with how busy the repository is, against the 5,000 requests an +// hour a token is allowed. Measured over this collector's own history: 4 active +// runs at the median, 43 at p95, 51 at peak. +// +// active runs at 60s at 30s +// 4 360/hr 720/hr +// 43 2,700/hr 5,400/hr +// 51 3,180/hr 6,360/hr +// +// A minute is comfortable across that range and doubles the resolution. Thirty +// seconds exceeds the budget above roughly forty concurrent runs — which is +// exactly when the queue is worth watching — so it needs the per-sample cost +// reduced first, not the interval shortened. +const DEFAULT_INTERVAL_SECONDS = 60; // Requests deliberately left unspent, so a burst of runs can never take the // token to zero and lock out other consumers. From ccfed5de78fa6a375f5a5e8ef3883385cc28ebd4 Mon Sep 17 00:00:00 2001 From: Buck Doyle Date: Thu, 20 Aug 2026 09:07:21 -0400 Subject: [PATCH 14/15] Trigger the collector deploy from this branch again, temporarily The running task predates the fixes on this branch: it still publishes a snapshot when a job listing fails, and it samples on the old interval. Neither reaches staging until the deploy runs, and workflow_dispatch will not appear until this file is on the default branch. Remove before merge, as last time. Co-Authored-By: Claude Opus 5 --- .github/workflows/manual-deploy-actions-collector.yml | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/.github/workflows/manual-deploy-actions-collector.yml b/.github/workflows/manual-deploy-actions-collector.yml index 22de91e89e2..1d737a750ce 100644 --- a/.github/workflows/manual-deploy-actions-collector.yml +++ b/.github/workflows/manual-deploy-actions-collector.yml @@ -14,6 +14,13 @@ on: description: Deployment environment required: false default: staging + # TEMPORARY — remove before merge. workflow_dispatch only works once the file + # is on the default branch, so until this merges there is no other way to + # deploy the collector, and the running task is on an older image than this + # branch. This trigger lets the fixes reach staging so the dashboard reflects + # them. + push: + branches: [cs-12571-make-the-github-actions-backlog-visible-in-grafana] permissions: contents: read From c54dd43476e0daf73f0c68c85037b191f0c23481 Mon Sep 17 00:00:00 2001 From: Buck Doyle Date: Thu, 20 Aug 2026 09:27:01 -0400 Subject: [PATCH 15/15] Return the collector deploy to dispatch only The temporary trigger has delivered the fixes to staging. Leaving it would redeploy on every push to this branch, and once merged the dispatch entry works on its own. Co-Authored-By: Claude Opus 5 --- .github/workflows/manual-deploy-actions-collector.yml | 7 ------- 1 file changed, 7 deletions(-) diff --git a/.github/workflows/manual-deploy-actions-collector.yml b/.github/workflows/manual-deploy-actions-collector.yml index 1d737a750ce..22de91e89e2 100644 --- a/.github/workflows/manual-deploy-actions-collector.yml +++ b/.github/workflows/manual-deploy-actions-collector.yml @@ -14,13 +14,6 @@ on: description: Deployment environment required: false default: staging - # TEMPORARY — remove before merge. workflow_dispatch only works once the file - # is on the default branch, so until this merges there is no other way to - # deploy the collector, and the running task is on an older image than this - # branch. This trigger lets the fixes reach staging so the dashboard reflects - # them. - push: - branches: [cs-12571-make-the-github-actions-backlog-visible-in-grafana] permissions: contents: read