Add Actions dashboard for Grafana - #5824
Merged
backspace merged 17 commits intoAug 20, 2026
Merged
Conversation
GitHub's Actions UI cannot say how deep the queue is, how long jobs have been waiting, or which branches and people are holding the runners. Establishing that during the recent archive outage meant scripting the REST API by hand, and the answer disappeared as soon as the terminal scrolled. A collector samples the Actions API and prints one JSON line per observation, in the same shape every service here already logs, so the existing Alloy and FireLens pipelines carry it to Loki without special handling. A dashboard in the boxel-status folder reads it back: queue depth and wait percentiles over time, runners consumed by branch, author and workflow, and a table of in-progress jobs ranked by how long their current step has run — the signal that separates a wedged job from a slow one. Grouped depth is emitted as its own lines rather than nested on the snapshot, because LogQL flattens nested JSON into one label per key, which for branch names yields a label per branch instead of a series grouped by branch. Two constraints shape the collector. It must not run as a scheduled Actions workflow, since it would queue behind the backlog it measures and go blind during the incident it exists for. And a sample costs one request per active run, so at the ~60 runs this repository sustains, a one-minute interval would consume most of a token's hourly REST budget; the interval is two minutes and a sample is skipped rather than reported partially when the remaining budget nears its reserve. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…uction The collector is a single file that needs nothing beyond the Node runtime and global fetch, so its image is the runtime plus that file — no package install and no lockfile. Deployment is workflow_dispatch only. Putting the deploy of a queue sampler on a CI trigger would make it queue behind the backlog it reports on, which is the same reason the collector itself does not run inside Actions. The hosted task reads its credential from ACTIONS_COLLECTOR_GITHUB_TOKEN rather than a bare GITHUB_TOKEN, because its SSM path is shared with every other boxel service and a generic name there would be ambiguous. The plain name still works for local runs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The token was documented as needing `actions: read`, which is wrong for this repository and misleading about why it exists. Both endpoints the collector reads answer unauthenticated on a public repository — the token's only job is the rate limit, which authentication raises from 60 requests an hour to 5,000, against a single sample that can cost sixty. So no permissions are needed while boxel is public. The private-repository requirement is stated separately, since that is when Actions: Read-only would start to matter. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
Observability diff (vs staging)Show diffdiff --git a/tmp/committed-canon.4eHpyb/dashboards/boxel-status/actions-queue.json b/tmp/committed-canon.4eHpyb/dashboards/boxel-status/actions-queue.json
new file mode 100644
index 0000000..9f6565f
--- /dev/null
+++ b/tmp/committed-canon.4eHpyb/dashboards/boxel-status/actions-queue.json
@@ -0,0 +1,1285 @@
+{
+ "apiVersion": "dashboard.grafana.app/v1beta1",
+ "kind": "Dashboard",
+ "metadata": {
+ "annotations": {
+ "grafana.app/folder": "defd2d156sav4d"
+ },
+ "name": "boxel-actions-queue"
+ },
+ "spec": {
+ "annotations": {
+ "list": []
+ },
+ "description": "GitHub Actions queue depth, wait times and runner consumption, sampled from the Actions REST API and shipped as JSON log lines on channel boxel:actions-queue.",
+ "editable": true,
+ "fiscalYearStartMonth": 0,
+ "graphTooltip": 0,
+ "links": [],
+ "panels": [
+ {
+ "collapsed": false,
+ "gridPos": {
+ "h": 1,
+ "w": 24,
+ "x": 0,
+ "y": 0
+ },
+ "id": 2,
+ "panels": [],
+ "title": "Queue depth",
+ "type": "row"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Jobs waiting for a runner at the most recent sample.",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {}
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 4,
+ "w": 4,
+ "x": 0,
+ "y": 1
+ },
+ "id": 3,
+ "interval": "5m",
+ "options": {
+ "colorMode": "value",
+ "graphMode": "area",
+ "justifyMode": "auto",
+ "orientation": "auto",
+ "reduceOptions": {
+ "calcs": [
+ "lastNotNull"
+ ],
+ "fields": "",
+ "values": false
+ },
+ "textMode": "auto",
+ "wideLayout": true
+ },
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "last_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"snapshot\" | unwrap queued_jobs [$__interval]) by ()",
+ "legendFormat": "queued",
+ "queryType": "range",
+ "refId": "A"
+ }
+ ],
+ "title": "Jobs queued now",
+ "type": "stat"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Jobs currently holding a runner.",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {}
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 4,
+ "w": 4,
+ "x": 4,
+ "y": 1
+ },
+ "id": 4,
+ "interval": "5m",
+ "options": {
+ "colorMode": "value",
+ "graphMode": "area",
+ "justifyMode": "auto",
+ "orientation": "auto",
+ "reduceOptions": {
+ "calcs": [
+ "lastNotNull"
+ ],
+ "fields": "",
+ "values": false
+ },
+ "textMode": "auto",
+ "wideLayout": true
+ },
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "last_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"snapshot\" | unwrap running_jobs [$__interval]) by ()",
+ "legendFormat": "running",
+ "queryType": "range",
+ "refId": "A"
+ }
+ ],
+ "title": "Jobs running now",
+ "type": "stat"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Workflow runs in queued or in_progress state.",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {}
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 4,
+ "w": 4,
+ "x": 8,
+ "y": 1
+ },
+ "id": 5,
+ "interval": "5m",
+ "options": {
+ "colorMode": "value",
+ "graphMode": "area",
+ "justifyMode": "auto",
+ "orientation": "auto",
+ "reduceOptions": {
+ "calcs": [
+ "lastNotNull"
+ ],
+ "fields": "",
+ "values": false
+ },
+ "textMode": "auto",
+ "wideLayout": true
+ },
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "last_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"snapshot\" | unwrap active_runs [$__interval]) by ()",
+ "legendFormat": "runs",
+ "queryType": "range",
+ "refId": "A"
+ }
+ ],
+ "title": "Active runs",
+ "type": "stat"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Queued depth rising while running stays flat means the pool is saturated rather than slow.\n\nCarries the same three values as the stat tiles beside it, because a stat panel has no hover tooltip — read the headline there, hover here.",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {
+ "spanNulls": true
+ }
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 12,
+ "y": 1
+ },
+ "id": 6,
+ "interval": "5m",
+ "options": {
+ "legend": {
+ "calcs": [
+ "lastNotNull",
+ "max"
+ ],
+ "displayMode": "table",
+ "placement": "bottom",
+ "showLegend": true,
+ "sortBy": "Last *",
+ "sortDesc": true
+ },
+ "tooltip": {
+ "mode": "multi",
+ "sort": "desc"
+ }
+ },
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"snapshot\" | unwrap queued_jobs [$__interval]) by ()",
+ "legendFormat": "queued",
+ "queryType": "range",
+ "refId": "A"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"snapshot\" | unwrap running_jobs [$__interval]) by ()",
+ "legendFormat": "running",
+ "queryType": "range",
+ "refId": "B"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"snapshot\" | unwrap active_runs [$__interval]) by ()",
+ "legendFormat": "active runs",
+ "queryType": "range",
+ "refId": "C"
+ }
+ ],
+ "title": "Queue depth over time",
+ "type": "timeseries"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Age of the oldest job still waiting for a runner.",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {},
+ "unit": "s"
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 4,
+ "w": 12,
+ "x": 0,
+ "y": 5
+ },
+ "id": 7,
+ "interval": "5m",
+ "options": {
+ "colorMode": "value",
+ "graphMode": "area",
+ "justifyMode": "auto",
+ "orientation": "auto",
+ "reduceOptions": {
+ "calcs": [
+ "lastNotNull"
+ ],
+ "fields": "",
+ "values": false
+ },
+ "textMode": "auto",
+ "wideLayout": true
+ },
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"job\" | status=\"queued\" | unwrap queued_seconds [$__interval]) by ()",
+ "legendFormat": "longest wait",
+ "queryType": "range",
+ "refId": "A"
+ }
+ ],
+ "title": "Longest current wait",
+ "type": "stat"
+ },
+ {
+ "collapsed": false,
+ "gridPos": {
+ "h": 1,
+ "w": 24,
+ "x": 0,
+ "y": 18
+ },
+ "id": 8,
+ "panels": [],
+ "title": "Wait times",
+ "type": "row"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "How long jobs have been waiting, measured from job creation.",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {
+ "spanNulls": true
+ },
+ "unit": "s"
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 24,
+ "x": 0,
+ "y": 19
+ },
+ "id": 9,
+ "interval": "5m",
+ "options": {
+ "legend": {
+ "calcs": [
+ "lastNotNull",
+ "max"
+ ],
+ "displayMode": "table",
+ "placement": "bottom",
+ "showLegend": true,
+ "sortBy": "Last *",
+ "sortDesc": true
+ },
+ "tooltip": {
+ "mode": "multi",
+ "sort": "desc"
+ }
+ },
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "quantile_over_time(0.5, {service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"job\" | status=\"queued\" | unwrap queued_seconds [$__interval]) by ()",
+ "legendFormat": "p50",
+ "queryType": "range",
+ "refId": "A"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "quantile_over_time(0.9, {service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"job\" | status=\"queued\" | unwrap queued_seconds [$__interval]) by ()",
+ "legendFormat": "p90",
+ "queryType": "range",
+ "refId": "B"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"job\" | status=\"queued\" | unwrap queued_seconds [$__interval]) by ()",
+ "legendFormat": "max",
+ "queryType": "range",
+ "refId": "C"
+ }
+ ],
+ "title": "Queue wait percentiles",
+ "type": "timeseries"
+ },
+ {
+ "collapsed": false,
+ "gridPos": {
+ "h": 1,
+ "w": 24,
+ "x": 0,
+ "y": 27
+ },
+ "id": 10,
+ "panels": [],
+ "title": "Who is consuming the pool",
+ "type": "row"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Queued jobs grouped by branch, from the collector's per-group snapshot lines.",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {
+ "spanNulls": true
+ }
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 8,
+ "x": 0,
+ "y": 28
+ },
+ "id": 11,
+ "interval": "5m",
+ "options": {
+ "legend": {
+ "calcs": [
+ "lastNotNull",
+ "max"
+ ],
+ "displayMode": "table",
+ "placement": "bottom",
+ "showLegend": true,
+ "sortBy": "Last *",
+ "sortDesc": true
+ },
+ "tooltip": {
+ "mode": "multi",
+ "sort": "desc"
+ }
+ },
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "topk(10, sum by (key) (max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"group\" | dimension=\"branch\" | unwrap queued [$__interval])))",
+ "legendFormat": "{{key}}",
+ "queryType": "range",
+ "refId": "A"
+ }
+ ],
+ "title": "Queued jobs by branch",
+ "type": "timeseries"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Queued jobs grouped by actor, from the collector's per-group snapshot lines.",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {
+ "spanNulls": true
+ }
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 8,
+ "x": 8,
+ "y": 28
+ },
+ "id": 12,
+ "interval": "5m",
+ "options": {
+ "legend": {
+ "calcs": [
+ "lastNotNull",
+ "max"
+ ],
+ "displayMode": "table",
+ "placement": "bottom",
+ "showLegend": true,
+ "sortBy": "Last *",
+ "sortDesc": true
+ },
+ "tooltip": {
+ "mode": "multi",
+ "sort": "desc"
+ }
+ },
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "topk(10, sum by (key) (max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"group\" | dimension=\"actor\" | unwrap queued [$__interval])))",
+ "legendFormat": "{{key}}",
+ "queryType": "range",
+ "refId": "A"
+ }
+ ],
+ "title": "Queued jobs by author",
+ "type": "timeseries"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Queued jobs grouped by workflow, from the collector's per-group snapshot lines.",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {
+ "spanNulls": true
+ }
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 8,
+ "x": 16,
+ "y": 28
+ },
+ "id": 13,
+ "interval": "5m",
+ "options": {
+ "legend": {
+ "calcs": [
+ "lastNotNull",
+ "max"
+ ],
+ "displayMode": "table",
+ "placement": "bottom",
+ "showLegend": true,
+ "sortBy": "Last *",
+ "sortDesc": true
+ },
+ "tooltip": {
+ "mode": "multi",
+ "sort": "desc"
+ }
+ },
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "topk(10, sum by (key) (max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"group\" | dimension=\"workflow\" | unwrap queued [$__interval])))",
+ "legendFormat": "{{key}}",
+ "queryType": "range",
+ "refId": "A"
+ }
+ ],
+ "title": "Queued jobs by workflow",
+ "type": "timeseries"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Runners currently held, grouped by branch.",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {
+ "spanNulls": true
+ }
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 8,
+ "x": 0,
+ "y": 36
+ },
+ "id": 14,
+ "interval": "5m",
+ "options": {
+ "legend": {
+ "calcs": [
+ "lastNotNull",
+ "max"
+ ],
+ "displayMode": "table",
+ "placement": "bottom",
+ "showLegend": true,
+ "sortBy": "Last *",
+ "sortDesc": true
+ },
+ "tooltip": {
+ "mode": "multi",
+ "sort": "desc"
+ }
+ },
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "topk(10, sum by (key) (max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"group\" | dimension=\"branch\" | unwrap running [$__interval])))",
+ "legendFormat": "{{key}}",
+ "queryType": "range",
+ "refId": "A"
+ }
+ ],
+ "title": "Running jobs by branch",
+ "type": "timeseries"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Runners currently held, grouped by actor.",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {
+ "spanNulls": true
+ }
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 8,
+ "x": 8,
+ "y": 36
+ },
+ "id": 15,
+ "interval": "5m",
+ "options": {
+ "legend": {
+ "calcs": [
+ "lastNotNull",
+ "max"
+ ],
+ "displayMode": "table",
+ "placement": "bottom",
+ "showLegend": true,
+ "sortBy": "Last *",
+ "sortDesc": true
+ },
+ "tooltip": {
+ "mode": "multi",
+ "sort": "desc"
+ }
+ },
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "topk(10, sum by (key) (max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"group\" | dimension=\"actor\" | unwrap running [$__interval])))",
+ "legendFormat": "{{key}}",
+ "queryType": "range",
+ "refId": "A"
+ }
+ ],
+ "title": "Running jobs by author",
+ "type": "timeseries"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Runners currently held, grouped by workflow.",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {
+ "spanNulls": true
+ }
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 8,
+ "x": 16,
+ "y": 36
+ },
+ "id": 16,
+ "interval": "5m",
+ "options": {
+ "legend": {
+ "calcs": [
+ "lastNotNull",
+ "max"
+ ],
+ "displayMode": "table",
+ "placement": "bottom",
+ "showLegend": true,
+ "sortBy": "Last *",
+ "sortDesc": true
+ },
+ "tooltip": {
+ "mode": "multi",
+ "sort": "desc"
+ }
+ },
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "topk(10, sum by (key) (max_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"group\" | dimension=\"workflow\" | unwrap running [$__interval])))",
+ "legendFormat": "{{key}}",
+ "queryType": "range",
+ "refId": "A"
+ }
+ ],
+ "title": "Running jobs by workflow",
+ "type": "timeseries"
+ },
+ {
+ "collapsed": false,
+ "gridPos": {
+ "h": 1,
+ "w": 24,
+ "x": 0,
+ "y": 44
+ },
+ "id": 17,
+ "panels": [],
+ "title": "Long-running steps",
+ "type": "row"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Ranked by how long a job's current step has been running. A job whose step has run far longer than the same step on its siblings is the wedge signal — compare before concluding it is stuck.\n\nThe window is four minutes against a two-minute sample interval, so a row can be up to that stale: a job that finished moments ago may still appear, and a job that just changed step may briefly show both. Neither distorts what this panel is for, since a stall worth acting on is measured in hours.",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {},
+ "unit": "s"
+ },
+ "overrides": [
+ {
+ "matcher": {
+ "id": "byName",
+ "options": "job"
+ },
+ "properties": [
+ {
+ "id": "links",
+ "value": [
+ {
+ "targetBlank": true,
+ "title": "Open this job on GitHub",
+ "url": "https://github.com/cardstack/boxel/actions/runs/${__data.fields.run_id}/job/${__data.fields.job_id}"
+ }
+ ]
+ }
+ ]
+ },
+ {
+ "matcher": {
+ "id": "byName",
+ "options": "head_branch"
+ },
+ "properties": [
+ {
+ "id": "links",
+ "value": [
+ {
+ "targetBlank": true,
+ "title": "Actions runs for this branch",
+ "url": "https://github.com/cardstack/boxel/actions?query=branch%3A${__value.raw}"
+ }
+ ]
+ }
+ ]
+ },
+ {
+ "matcher": {
+ "id": "byName",
+ "options": "actor"
+ },
+ "properties": [
+ {
+ "id": "links",
+ "value": [
+ {
+ "targetBlank": true,
+ "title": "GitHub profile",
+ "url": "https://github.com/${__value.raw}"
+ }
+ ]
+ }
+ ]
+ }
+ ]
+ },
+ "gridPos": {
+ "h": 10,
+ "w": 24,
+ "x": 0,
+ "y": 45
+ },
+ "id": 18,
+ "options": {},
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "topk(25, max by (job, head_branch, workflow, actor, current_step, run_id, job_id) (last_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"job\" | status=\"in_progress\" | unwrap current_step_seconds [4m])))",
+ "queryType": "instant",
+ "refId": "A"
+ }
+ ],
+ "title": "In-progress jobs by current step duration",
+ "transformations": [
+ {
+ "id": "organize",
+ "options": {
+ "excludeByName": {
+ "Time": true,
+ "job_id": true,
+ "run_id": true
+ },
+ "renameByName": {
+ "Value": "in step",
+ "Value #A": "in step"
+ }
+ }
+ }
+ ],
+ "type": "table"
+ },
+ {
+ "collapsed": false,
+ "gridPos": {
+ "h": 1,
+ "w": 24,
+ "x": 0,
+ "y": 55
+ },
+ "id": 19,
+ "panels": [],
+ "title": "Collector health",
+ "type": "row"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Sampling stops rather than reporting partial depth when this nears the reserve.",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {
+ "spanNulls": true
+ }
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 6,
+ "w": 12,
+ "x": 0,
+ "y": 56
+ },
+ "id": 20,
+ "interval": "5m",
+ "options": {
+ "legend": {
+ "calcs": [
+ "lastNotNull",
+ "max"
+ ],
+ "displayMode": "table",
+ "placement": "bottom",
+ "showLegend": true,
+ "sortBy": "Last *",
+ "sortDesc": true
+ },
+ "tooltip": {
+ "mode": "multi",
+ "sort": "desc"
+ }
+ },
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "min_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"snapshot\" | unwrap rate_limit_remaining [$__interval]) by ()",
+ "legendFormat": "remaining",
+ "queryType": "range",
+ "refId": "A"
+ }
+ ],
+ "title": "GitHub API requests remaining",
+ "type": "timeseries"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Gaps in the series above should have a matching line here.",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {}
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 6,
+ "w": 12,
+ "x": 12,
+ "y": 56
+ },
+ "id": 21,
+ "options": {},
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "{service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=~\"collector-error|collector-throttled\"",
+ "queryType": "range",
+ "refId": "A"
+ }
+ ],
+ "title": "Collector errors and throttling",
+ "type": "logs"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "fieldConfig": {
+ "defaults": {
+ "custom": {}
+ },
+ "overrides": []
+ },
+ "gridPos": {
+ "h": 4,
+ "w": 24,
+ "x": 0,
+ "y": 62
+ },
+ "id": 1,
+ "options": {
+ "content": "Sampled from the GitHub Actions REST API by `packages/observability/collectors/actions-queue.ts`, which emits one JSON line per observation on channel `boxel:actions-queue`.\n\nThe collector runs **outside** GitHub Actions on purpose — a scheduled workflow would queue behind the backlog it measures and go blind during the incident it exists for.\n\nDepth and grouped depth come from periodic snapshots, so each point is a sample rather than a continuous measure; gaps mean the collector was down or throttled (see *Collector health*).",
+ "mode": "markdown"
+ },
+ "targets": [],
+ "title": "About this dashboard",
+ "type": "text"
+ },
+ {
+ "collapsed": false,
+ "gridPos": {
+ "h": 1,
+ "w": 24,
+ "x": 0,
+ "y": 9
+ },
+ "id": 40,
+ "panels": [],
+ "title": "Right now",
+ "type": "row"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "A point-in-time count, unlike the timeseries above — a branch whose jobs have finished disappears from this table instead of leaving a line behind.",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {}
+ },
+ "overrides": [
+ {
+ "matcher": {
+ "id": "byName",
+ "options": "branch"
+ },
+ "properties": [
+ {
+ "id": "links",
+ "value": [
+ {
+ "targetBlank": true,
+ "title": "Actions runs for this branch",
+ "url": "https://github.com/cardstack/boxel/actions?query=branch%3A${__value.raw}"
+ }
+ ]
+ }
+ ]
+ }
+ ]
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 8,
+ "x": 0,
+ "y": 10
+ },
+ "id": 41,
+ "interval": "5m",
+ "options": {},
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "sum by (key) (last_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"group\" | dimension=\"branch\" | unwrap queued [5m])) > 0",
+ "queryType": "instant",
+ "refId": "A"
+ }
+ ],
+ "title": "Queued now, by branch",
+ "transformations": [
+ {
+ "id": "organize",
+ "options": {
+ "excludeByName": {
+ "Time": true
+ },
+ "renameByName": {
+ "Value": "jobs",
+ "Value #A": "jobs",
+ "key": "branch"
+ }
+ }
+ },
+ {
+ "id": "sortBy",
+ "options": {
+ "fields": {},
+ "sort": [
+ {
+ "desc": true,
+ "field": "jobs"
+ }
+ ]
+ }
+ }
+ ],
+ "type": "table"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "Which branches are occupying the pool at this moment.",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {}
+ },
+ "overrides": [
+ {
+ "matcher": {
+ "id": "byName",
+ "options": "branch"
+ },
+ "properties": [
+ {
+ "id": "links",
+ "value": [
+ {
+ "targetBlank": true,
+ "title": "Actions runs for this branch",
+ "url": "https://github.com/cardstack/boxel/actions?query=branch%3A${__value.raw}"
+ }
+ ]
+ }
+ ]
+ }
+ ]
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 8,
+ "x": 8,
+ "y": 10
+ },
+ "id": 42,
+ "interval": "5m",
+ "options": {},
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "sum by (key) (last_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"group\" | dimension=\"branch\" | unwrap running [5m])) > 0",
+ "queryType": "instant",
+ "refId": "A"
+ }
+ ],
+ "title": "Runners held now, by branch",
+ "transformations": [
+ {
+ "id": "organize",
+ "options": {
+ "excludeByName": {
+ "Time": true
+ },
+ "renameByName": {
+ "Value": "jobs",
+ "Value #A": "jobs",
+ "key": "branch"
+ }
+ }
+ },
+ {
+ "id": "sortBy",
+ "options": {
+ "fields": {},
+ "sort": [
+ {
+ "desc": true,
+ "field": "jobs"
+ }
+ ]
+ }
+ }
+ ],
+ "type": "table"
+ },
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "description": "The same instant count grouped by whoever triggered the run.",
+ "fieldConfig": {
+ "defaults": {
+ "custom": {}
+ },
+ "overrides": [
+ {
+ "matcher": {
+ "id": "byName",
+ "options": "author"
+ },
+ "properties": [
+ {
+ "id": "links",
+ "value": [
+ {
+ "targetBlank": true,
+ "title": "GitHub profile",
+ "url": "https://github.com/${__value.raw}"
+ }
+ ]
+ }
+ ]
+ }
+ ]
+ },
+ "gridPos": {
+ "h": 8,
+ "w": 8,
+ "x": 16,
+ "y": 10
+ },
+ "id": 43,
+ "interval": "5m",
+ "options": {},
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "editorMode": "code",
+ "expr": "sum by (key) (last_over_time({service=\"actions-collector\", env=\"$env\"} |= \"boxel:actions-queue\" | json | line_format \"{{ if .log }}{{ .log }}{{ else }}{{ __line__ }}{{ end }}\" | json | channel=\"boxel:actions-queue\" | event_type=\"group\" | dimension=\"actor\" | unwrap queued [5m])) > 0",
+ "queryType": "instant",
+ "refId": "A"
+ }
+ ],
+ "title": "Queued now, by author",
+ "transformations": [
+ {
+ "id": "organize",
+ "options": {
+ "excludeByName": {
+ "Time": true
+ },
+ "renameByName": {
+ "Value": "jobs",
+ "Value #A": "jobs",
+ "key": "author"
+ }
+ }
+ },
+ {
+ "id": "sortBy",
+ "options": {
+ "fields": {},
+ "sort": [
+ {
+ "desc": true,
+ "field": "jobs"
+ }
+ ]
+ }
+ }
+ ],
+ "type": "table"
+ }
+ ],
+ "refresh": "1m",
+ "schemaVersion": 42,
+ "tags": [
+ "ci",
+ "actions",
+ "forensics"
+ ],
+ "templating": {
+ "list": [
+ {
+ "hide": 2,
+ "name": "env",
+ "query": "__ENV__",
+ "skipUrlSync": true,
+ "type": "constant"
+ }
+ ]
+ },
+ "time": {
+ "from": "now-6h",
+ "to": "now"
+ },
+ "timepicker": {},
+ "timezone": "browser",
+ "title": "GitHub Actions Queue",
+ "weekStart": ""
+ }
+}
(Run: https://github.com/cardstack/boxel/actions/runs/32374347665) |
…hub-actions-backlog-visible-in-grafana
A workflow_dispatch workflow can only be dispatched once its file is on the default branch, so until this merges there is no way to build the collector — and with no collector running, the dashboard previews on this PR with every panel empty, which is indistinguishable from a query that is simply wrong. A push trigger scoped to this branch closes that loop, letting the queries be checked against real data before merging. The staging environment carries no branch policy and the OIDC trust policy for the deploy role admits any ref of this repository, so a branch build is permitted. `inputs.environment` is null on a push event, so the environment default is now explicit at each use — worth keeping after the trigger goes. Remove the push trigger before merge. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
GitHub returns 403 for a rejected credential, for a token whose permissions do not cover the resource, and for an exhausted rate limit alike, distinguishing them only in the response body. The emitted error carried just the status, so a collector sitting at 403 said nothing about which of those it was. Carrying a bounded slice of the body and the remaining request budget into the error puts that in the collector-error line, where the dashboard already shows it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every metric panel drew one series per sample rather than one line. `| json` promotes each field of a record to a label, so successive samples never shared a label set and a range aggregation without an explicit `by ()` grouped none of them together. The stat tiles showed a row of numbers instead of a value, and the panels reading per-job fields returned so many series they rendered as no data at all. Every one of these queries has now been run against staging Loki: each returns a single series, and the grouped panels return one per group. The long-running steps table gains the actor, since the first question asked of a job that has sat in one step for hours is whose it is. The temporary push trigger goes with them, having served its purpose — the collector is deployed and reporting, which is what let these queries be checked against real data rather than merged on the assumption they were right. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…hub-actions-backlog-visible-in-grafana
The grouped panels drew a series per branch and listed them alphabetically, so the branch holding most of the queue could sit below the fold while smaller ones were named above it — the panel answered "which branches have queued jobs" when the question being asked of it is "which branch has the most". The legend is now a table sorted by last value, showing last and max, and each grouped query is bounded to the ten largest series at any instant so a busy hour cannot push the ones that matter out of view. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The collector samples every two minutes, but Grafana derives $__interval from panel width and on a multi-hour window that lands well under the sample period. Most range windows therefore covered no sample at all — at thirty seconds the queries return nothing — so the panels rendered as isolated spikes rather than a series, and the wait panels in particular looked like noise. Flooring the panel interval at five minutes keeps every window covering at least one sample, and spanning nulls bridges a gap where a sample is missed rather than breaking the line. Every query was re-run against staging Loki at the floored interval: each returns a continuous series where it previously returned a handful of scattered points. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The description occupied the top of the page, so opening the dashboard during an incident meant scrolling past prose to reach the queue depth. It reads better as an appendix: useful once, when someone wants to know where the numbers come from, and in the way every other time. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The grouped panels are timeseries over the dashboard's window, so a branch that was queue-heavy an hour ago keeps a line long after its jobs have gone. Read at a glance during an incident that says "this branch has the most queued", when what it means is "this branch had the most queued at some point in the last six hours" — twice now that has sent someone looking for jobs that had already finished. A "Right now" row of instant queries answers the other question directly: what is queued, and who is holding runners, at this moment. Rows with nothing queued are filtered out, so a branch drops off the table when its work is done rather than lingering at zero. The timeseries stay. "How did we get here" and "what is true now" are different questions and the history is the right shape for the first. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Seeing that a job has sat in one step for four hours raises exactly one question, and answering it meant copying a job name into the Actions search by hand. The rows now carry links on the text already in them, so no column shows a URL. A job opens its own page on GitHub, which needs the run and job identifiers, so the query carries them and the transformation hides the columns — they are identifiers rather than information. A branch opens the Actions runs filtered to it, and an author their profile. Every link was resolved against a live row and fetched; all six return 200. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
backspace
marked this pull request as ready for review
August 20, 2026 12:15
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e041fbf6a2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
A failed job listing for one active run was logged and skipped, and the grouped totals and snapshot were then emitted regardless. That put a confidently low queue depth on the dashboard at exactly the moment the API was failing — contradicting the rate-limit reserve a few lines above, which abandons a sample for precisely that reason. Now only a 404 continues: a run reaped between the listing and the fetch has no jobs left to count, so omitting it understates nothing. Any other status means this run's jobs are unknown rather than absent, so the sample is abandoned with a collector-aborted line, leaving a visible gap instead of a wrong number. The long-running-steps table narrows its window from ten minutes to four, which bounds how long a finished job's last observation can linger in a table about what is running now. Its description states the residual, because a job that changes step mid-window still contributes a row per step and shortening the window measurably does not change that. Queue depth over time also carries active runs, so each stat tile beside it has a hoverable counterpart — a stat panel has no tooltip of its own. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Measured over the collector's own history the budget is nowhere near spent: 4 active runs at the median, 43 at p95, 51 at peak, and the remaining request count never fell below 3,792 of 5,000. A minute doubles the resolution and spends about 3,200 an hour at that peak. Thirty seconds does not fit. The cost is one request per active run, so it crosses the hourly limit somewhere around forty concurrent runs — which is precisely when the queue is worth watching closely. The reserve would keep the token alive by skipping samples, meaning the collector would go sparse exactly during an incident. Going finer than a minute needs the per-sample cost reduced, not the interval shortened. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The running task predates the fixes on this branch: it still publishes a snapshot when a job listing fails, and it samples on the old interval. Neither reaches staging until the deploy runs, and workflow_dispatch will not appear until this file is on the default branch. Remove before merge, as last time. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The temporary trigger has delivered the fixes to staging. Leaving it would redeploy on every push to this branch, and once merged the dispatch entry works on its own. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
lukemelia
approved these changes
Aug 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This makes it clearer what and whose jobs and runs are queued and in progress. There’s a table of long-running steps which can clarify whether there are systemic issues.