Fix swarm transport walls: out-of-band prompt, telemetry, lens split

.claude-plugin/marketplace.json

@@ -30,7 +30,7 @@
       "name": "swarm",
       "source": "./plugins/swarm",
       "description": "Local mixture-of-agents code review for Claude Code. Fans a diff across Claude lenses plus the codex and grok CLIs (grok-4.5) — every voice running one call per gated lens cluster — with file-read + hardened web research under an OS secret-jail, merges by mechanism with cross-family consensus, verifies solo findings and all design suggestions, and presents one ranked report. Optional --fix / --loop applies the findings you agreed with; --pr reviews a GitHub PR diff and posts the result. Skills: /swarm:review, /swarm:agents.",
-      "version": "0.7.0"
+      "version": "0.9.2"
     },
     {
       "name": "settings",

.claude/knowledge/_index.md

@@ -18,7 +18,7 @@
 - `features/herdr-tab-glyphs.md` — Task-state glyphs (`○ ● ◇ ◆ ✓`) + main-root `◉` on herdr tab labels: `states` mode in the self-contained renderer, sync-vs-`--cached` PR refresh per caller, exact-cwd rename rules, soft pr-flow shim
 - `features/kickoff-agent-selection.md` — `/kickoff` worker choice: single committed per-repo default (no global/fallback/ranking) else picker; `agent-registry.sh` as SoT; bounded model-aware grok/kimi probes (inconclusive→trust-auth); kimi's two-phase seed+continue argv + `argv_shell=`; non-claude "document, don't fake" degradation; announce-not-prompt for external defaults
 - `features/task-archiving-on-close.md` — `/close` archives (not deletes) the task file; adaptive commit + ff-push to main; per-repo `.claude/work-system-close-autocommit` opt-in skips the ask
-- `features/swarm-backend-adapter.md` — 0.6.0 read+web posture: OS secret-jail (denylist, worktree-aware, git-config-safe), per-voice fail-closed degrade, `jail` verb, prompt egress guard + residual risks; plus verified codex/grok CLI facts (schema JSON, effort mapping, model-aware readiness)
+- `features/swarm-backend-adapter.md` — 0.6.0 read+web posture: OS secret-jail (denylist, worktree-aware, git-config-safe), per-voice fail-closed degrade, `jail` verb, prompt egress guard + residual risks; plus verified codex/grok CLI facts (out-of-band prompt transport vs. the argv/`MAX_ARG_STRLEN` wall, schema JSON, effort mapping, model-aware readiness); measured runtime drivers (cluster 13x > effort 2.3x > size) + per-call telemetry
 - `features/swarm-review-pipeline.md` — `/swarm:review` pipeline: skill↔Workflow wiring, family-consensus, 0.5.0 lens clusters + design-kind verify, `--fix`/`--loop` (deterministic close-out via `loop-closeout.py`), `--pr` publish via deterministic `pr-post.py`
 
 ## Deployment

.claude/knowledge/features/swarm-backend-adapter.md

@@ -1,9 +1,9 @@
 ---
 title: "Swarm Backend Adapter Layer"
 createdAt: 2026-07-03
-updatedAt: 2026-07-23
+updatedAt: 2026-08-06
 createdFrom: "PR #21"
-updatedFrom: "open-swarm-external-exploration"
+updatedFrom: "fix-swarm-timeout-ceiling"
 pluginVersion: 1.9.0
 prime: false
 reindexedAt: 2026-07-12
@@ -129,8 +129,32 @@ The 120-KiB inline-diff cap is **unchanged** in 0.6.0; file-read now makes a
 future reduction of inlining possible (have the agent read the file itself) —
 coordinate that separately, do not duplicate transport work here.
 
-## Verified CLI facts (codex 0.144.6 / grok 0.2.103, 2026-07)
+## Verified CLI facts (codex 0.144.6 / grok 0.2.112, 2026-07..08)
 
+- **The prompt travels OUT-OF-BAND, never on argv** — codex reads it from stdin
+  (`-- -`; the help states an omitted or `-` PROMPT reads stdin), grok takes
+  `--prompt-file <path>` (present on 0.2.112; the introducing release is not
+  documented, so the adapter probes `grok --help` for the flag rather than
+  parsing a version). *Why it matters:* on argv the binding limit is
+  `MAX_ARG_STRLEN` (128 KiB on Linux), which forced a 120 KiB prompt cap — and
+  above that cap `/swarm:review` dropped **every** external voice, i.e. the same
+  damage as a backend timeout, from a size limit that was never inherent to the
+  backends. What remains is a model-context sanity cap
+  (`SWARM_MAX_PROMPT_BYTES`, default 512 KiB), read by the adapter AND the
+  skill's oversize guard from the same env knob so an override reaches both.
+  Verified end-to-end at 164 KiB through both backends (2026-08-05).
+  - **Do not "solve" a size limit by having the backend read the diff file
+    itself.** Both voices have file-read, so it looks equivalent — it is not:
+    delivery stops being verifiable (a model that reads only the file's head
+    silently loses coverage), the untrusted diff arrives as a tool result
+    instead of inside the nonce fence, and every voice pays an extra
+    round-trip. Out-of-band transport keeps the fence and the delivery
+    guarantee intact.
+  - grok reads that file from **inside the OS jail**, so it must be
+    jail-readable — `TMPDIR` is (the denylist covers credential paths). The
+    adapter's own temp prompt is `chmod 600` before content lands and is removed
+    by the EXIT trap on every path, including errors: it holds the untrusted
+    diff.
 - **Uniform findings JSON** is achievable from both CLIs: `codex exec
   --output-schema <file>` and `grok --json-schema '<inline>'` both enforce a
   JSON Schema on the final answer. One bundled schema
@@ -142,13 +166,37 @@ coordinate that separately, do not duplicate transport work here.
   `--output-last-message <file>` (stdout carries the agent transcript,
   stderr the progress log); grok prints a response **envelope** on stdout —
   the validated object is its `.structuredOutput` field.
-- **The adapter pins `-m grok-4.5`** — the schema-capable model, and since
-  swarm 0.4.3 the *only* grok model it supports. grok 0.2.101 renamed it from
-  `grok-build` (same upstream pin-rename class as codex's `gpt-5.6-terra`;
-  verified drop-in: identical envelope/`structuredOutput` shape, `--single`
-  unchanged). Any other `--model` is preflight-rejected with a usage error —
-  only grok-4.5 enforces `--json-schema`, and an unlisted model fails late with
-  `structuredOutput: null` after burning a full review.
+- **The grok model is DISCOVERED, not pinned** (0.9.2). The adapter selects the
+  newest canonical id the CLI lists whose `--json-schema` enforcement is
+  *verified*; `GROK_DEFAULT_MODEL` is only the fallback floor. Ported from
+  `~/dotfiles`' `cc-harness-agents`, which tracks the same provider, with one
+  gate substituted: that helper withholds an upgrade until a model's context
+  window is known, the adapter until its SCHEMA ENFORCEMENT is known — a model
+  that merely accepts the flag and returns `structuredOutput: null` fails late,
+  after a full review is paid for.
+  - `GROK_CANONICAL_RE` accepts only **bare version ids, major ≥ 4**. A provider
+    catalog mixes canonical releases with non-substitutes: dated snapshots,
+    reasoning/non-reasoning splits, multi-agent, build, composer, image/video.
+    Major ≥ 4 keeps a catalog that regresses to `grok-3*` from pulling the
+    ensemble backwards.
+  - Version order is **component-wise**, so `grok-4.20` beats `grok-4.6` — as a
+    decimal fraction it would lose, but the provider means the 20th minor
+    release and already ships 4.20-derived ids.
+  - `GROK_SCHEMA_VERIFIED` is the hard gate and the upgrade ritual: a newer
+    canonical model is **named on stderr, never selected**, so adopting it is a
+    one-line edit after a hand check. Verified 2026-08-16 on CLI 1.0.3:
+    grok-4.5 and grok-4.6 both return an envelope whose `.structuredOutput`
+    carries the schema's `findings`.
+  - Readiness asks "is ANY verified model on offer", matching what the run would
+    actually select. The old "is THIS id listed" form is what let the 1.0.3
+    marker change drop grok from every review.
+- **`grok models` output format has changed twice — parse it defensively.**
+  0.2.101 renamed `grok-build` → `grok-4.5`; **1.0.3 changed the bullet marker**
+  so only the DEFAULT keeps `*` and the rest use `-`. The `*`-only matcher then
+  reported "this CLI does not offer grok-4.5" for a CLI that offered it, and
+  grok — the third model family — vanished from every review, silently and with
+  no timeout involved. `test_grok_models.py` pins both formats against the
+  shipped awk program.
 - **Effort ladders**: grok is `low|medium|high` since 0.2.101 (the `max` tier
   is gone) → the adapter maps `xhigh`/`max`→`high`; codex has no `max` tier →
   map `max`→`xhigh` (`-c model_reasoning_effort=…`). Both mappings degrade a
@@ -258,12 +306,83 @@ coordinate that separately, do not duplicate transport work here.
   Re-verify the pinned ids when bumping the tested CLI version. Never fall
   back to a broad denylist that could admit a mutating tool.
 
+## What actually drives external-call runtime (measured 2026-08-11)
+
+The `grok × breakage` timeouts were long blamed on prompt size. **Measured, they
+are not.** Same 42 KB diff, same adapter, one variable at a time:
+
+| Backend | Effort | Cluster | Duration | Findings |
+|---------|--------|---------|----------|----------|
+| grok | high | breakage | **374 s** | 4 |
+| grok | low | breakage | **161 s** | 4 |
+| grok | high | consistency (style) | **28 s** | 6 |
+| codex | high | breakage | **104 s** | 2 |
+
+Control: a **164 KiB** prompt at `low` with no lens instruction returned in
+**20 s** (grok) / **8.6 s** (codex). Four times the bytes, a twentieth of the
+time.
+
+- **The cluster dominates — by 13x.** breakage vs. consistency at identical
+  effort: 374 s → 28 s. `breakage` holds `cross-file-trace` ("read the
+  neighboring repo files, not just the diff") and `removed-behavior`; both
+  *require* exploration, and the tool loop is the cost. Prompt bytes are noise
+  next to it.
+- **Effort is secondary — 2.3x** (374 s → 161 s) and in this sample it bought
+  **zero extra findings** (4 either way). Lowering grok's effort for the
+  breakage cluster is cheap headroom, not a quality trade — but on its own it
+  only moves 62% of the wall to 27%, it does not remove the wall.
+- **Backends are not interchangeable — 3.6x.** codex ran the same breakage
+  prompt in 104 s where grok took 374 s. That is *why* grok is the one that
+  reproducibly dies and codex never has: it is the slow voice on the expensive
+  cluster.
+- **Consequence for any fix:** chunking the *diff* addresses the one variable
+  measurement rules out. Splitting by *lens* was shipped in 0.9.0 as the `reach`
+  cluster — but measure what it actually bought before repeating the reasoning:
+  it bounds a timeout's cost to one lens instead of three and fixes real lens
+  crowd-out, yet the longest call only fell 374 s → 313 s (see
+  [[swarm-review-pipeline]] § lens set). **The two-lens `breakage` cluster still
+  costs 313 s**, so `cross-file-trace` is the priciest lens but nowhere near the
+  whole bill — no single lens split clears the 600 s wall on its own.
+- **Still the largest untried lever for RUNTIME: effort.** 374 s → 161 s (2.3x)
+  for the identical 4 findings. It does not isolate failures the way the split
+  does, but for pure headroom under the wall nothing else measured comes close.
+- **`grok --max-turns N` was measured and REJECTED — do not reach for it.** It
+  caps the tool loop, but the useful range is a cliff, not a dial:
+
+  | `--max-turns` | duration | findings |
+  |---|---|---|
+  | 10 | 10 s | **0** |
+  | 20 | 279 s | 4 |
+  | (unset) | 374 s | 4 |
+
+  At 20 it saves 25%; at 10 it returns nothing at all. Worse, the truncated run
+  exits **rc=0 with an empty findings array** — so the adapter and the whole
+  pipeline read it as "reviewed cleanly, found nothing" rather than as a
+  failure. A timeout at least lands in `backendErrors`; this silently deletes a
+  voice's coverage while the report still counts it as a voice that ran. The
+  safe N is also diff-dependent (what needs 20 here may need 30 elsewhere), so
+  any fixed value eventually lands on the wrong side of that cliff. If this is
+  ever revisited, it MUST be paired with an empty-findings-under-turn-cap check
+  that converts the truncation into a loud backend error.
+
+`agents.sh run --telemetry <file> --unit <name>` records this per call
+(duration, effective effort/model, prompt bytes, backend rc, `timed_out`, and
+the wall the call actually ran under), written from the EXIT trap so a timeout
+is recorded too. `scripts/telemetry-report.py` renders it and flags any
+**surviving** call at ≥60% of its wall — the case `backendErrors` structurally
+cannot show, because a voice that finished at 550 s and one that finished at
+20 s are both just "ok".
+
 ## Gotchas (found in E2E testing, fixed in the adapter)
 
-- **codex hangs on inherited stdin.** With an open non-TTY stdin, `codex exec`
-  waits for "additional input from stdin" *in addition to* the positional
-  prompt — in a background shell this hangs forever. Always call it with
-  `</dev/null` (the adapter does).
+- **codex hangs on inherited stdin *when the prompt is on argv*.** With a
+  positional prompt AND an open non-TTY stdin, `codex exec` waits for
+  "additional input from stdin" (it appends it as a `<stdin>` block) — in a
+  background shell that hangs forever. The rule is "never leave stdin dangling",
+  NOT "always `</dev/null`": since the prompt-transport rework the adapter
+  deliberately feeds the prompt ON stdin (`-- -`) and closes the dangling case
+  by construction — stdin is the prompt and hits EOF. Any call that keeps a
+  positional prompt still needs `</dev/null`.
 - **`set -u` + EXIT trap + `local`**: a trap like `trap 'rm -f "$out"' EXIT`
   referencing a function-`local` variable fires after the function returned —
   under `set -u` the script then dies with "unbound variable" and **exit 1

.claude/knowledge/features/swarm-review-pipeline.md

@@ -1,9 +1,9 @@
 ---
 title: "Swarm Review Pipeline (/swarm:review)"
 createdAt: 2026-07-08
-updatedAt: 2026-07-27
+updatedAt: 2026-08-12
 createdFrom: "PR #24"
-updatedFrom: "swarm-per-lens-externals"
+updatedFrom: "fix-swarm-timeout-ceiling"
 pluginVersion: 1.9.0
 prime: false
 reindexedAt: 2026-07-12
@@ -19,7 +19,7 @@ lenses ∥ codex ∥ grok-4.5 (see [swarm-backend-adapter](swarm-backend-adapter
 A fourth, `grok-composer-2.5-fast`, was removed in swarm 0.4.3 — the grok CLI
 dropped the model.
 
-## Lens set: 11 lenses in 4 clusters (swarm 0.5.0)
+## Lens set: 11 lenses in 5 clusters (0.5.0; `reach` split off 0.9.0)
 
 Grown from 5 topical lenses by importing `/code-review`'s other two
 decomposition axes — methodological (HOW to look) and design quality — all
@@ -29,13 +29,35 @@ truth** — every voice's fan-out units come from it, Claude and externals alike
 
 | cluster | lenses | guiding question |
 |---|---|---|
-| `breakage` | correctness, removed-behavior, cross-file-trace | what breaks? |
+| `breakage` | correctness, removed-behavior | what breaks? |
+| `reach` | cross-file-trace | what else does this touch? |
 | `threat` | security, adversarial | what's exploitable / which assumption fails? |
 | `design` | reuse, simplification, efficiency, altitude | is this good, maintainable code? |
 | `consistency` | style, conventions | does it fit the codebase? |
 
+- **`reach` is a deliberate ONE-lens cluster** (0.9.0), split out of `breakage`
+  on measurement. The reason is **lens crowd-out**, NOT runtime — the split was
+  proposed as a speed fix and the measurement corrected that:
+
+  | run | duration | findings |
+  |---|---|---|
+  | old: one 3-lens `breakage` call | 374 s | 4 — **three of them `cross-file-trace`** |
+  | new: `breakage` (correctness, removed-behavior) | 313 s | 4 — *none* of which the combined call reported |
+  | new: `reach` (cross-file-trace) | 126 s | 4 — ≈ the combined call's cross-file set |
+
+  One lens was consuming the call's attention while the diff-local lenses barely
+  reported; splitting recovered four findings (three confirmed real against this
+  repo, incl. a silent config-validation gap). **What it does not buy: speed.**
+  The longest single call drops only 374 → 313 s (16%) and TOTAL work rises to
+  439 s, so `cross-file-trace` is the most exploration-heavy lens but not the
+  sole cost — this alone does not clear the 600 s wall. Two further effects,
+  neither reachable by lowering effort: a timeout now costs ONE lens instead of
+  three (`correctness`/`removed-behavior` survive it), and — carrying no
+  MANDATORY lens — the gate may prune the whole call on a diff with no
+  cross-file surface, where the old layout kept it alive because `correctness`
+  held the cluster open. Cost when kept: one extra call per live backend.
 - **The cluster is the fan-out unit for EVERY voice** since 0.7.0 — Claude
-  finders (≤4) *and* codex/grok (one CLI call per gated cluster each);
+  finders (≤5) *and* codex/grok (one CLI call per gated cluster each);
   `--max` splits all of them to one call per lens (≤11 units → ≤22 external
   calls) — the granularity ladder is `--quick` (future) =
   one broad pass → default = per-cluster → `--max` = per-lens. The **gate
@@ -264,7 +286,7 @@ filled* — `gh pr diff <n>` (bare `--pr` resolves the current branch's PR via
 Externals no longer run ONE broad multi-lens review each: codex and grok fan out
 over the **same gated clusters** as the Claude finders (`unitsFor()` builds the
 units once; `externalUnits` reuses `finderUnits` whenever a gate ran, so the two
-sides cannot drift). Cost is `live-backends × units` — ≤2×4 default, ≤2×11 under
+sides cannot drift). Cost is `live-backends × units` — ≤2×5 default, ≤2×11 under
 `--max` — logged at fan-out, never silently capped.
 
 Decisions worth keeping: